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FOREWORD 

This  technical  report  is  the  result  of  a site  visit  by  the  consultants  to  Fort  Gordon, 
Georgia,  location  of  the  Army's  Project  ABACUS,  a 128  terminal  multlminicompUer 
training  system.  This  is  the  only  major  computerized  training  project  In  the  United 
States  Army. 

In  reading  these  comments  the  reader  should  be  aware  of  the  tact  that  at  that  time 
the  system  installed  on  site  consisted  of  only  one  display  controller,  32  terminals 
and  very  limited  software.  The  ultimate  system  will  consist  of  4 display  controllers, 
a systems  controller,  a data  base  controller,  128  terminals  and  much  more  ex- 
tensive software.  This  limitation  was  not,  however,  critical  to  the  accomplishment 
of  the  objectives  of  the  conference.  The  participants  were  requested  to  comment 
primarily  on  the  long-range  implication  and  on  the  current  concept.  As  can  be  seen 
by  the  contents  of  this  report,  the  reactions  were  enthusiastic  and  the  remarks  quite 
candid.  To  assist  in  the  implementation  of  some  of  the  ideas  expressed,  I asked  my 
staff  to  prepare  an  executive  summary  of  the  recoramondations.  The  result  was  a 
24  page  document.  The  message  in  that  summary  came  through  loud  and  clear. 

The  Army  must  remain  flexible  in  all  areas  whether  it  be  hardware,  software, 
courseware  or  evaluation.  At  some  point  in  time,  however,  the  moment  of  decision 
must  be  reached. 

With  respect  to  hardware,  there  was  a wide  variety  of  opinions  varying  from  the 
current  trend  toward  stand-alone  terminals  to  a preference  for  large  regional 
systems.  The  concensus,  however,  was  that  each  requirement  must  be  met  on  its 
own  value  and  that  systems  should  be  tailored  to  meet  the  real  training  needs. 

Software  was  a major  concern  among  the  majority  of  the  consultants.  In  particular, 
the  applications  software,  and  the  language  to  be  used  for  authoring  was  subject  to 
question.  This  matter  has  already  been  addressed  at  the  Department  of  the  Army 
level,  our  needs  have  been  recognized  and  are  being  taken  into  consideration  by  a 
special  working  group  on  higher-order  languages.  When  considering  courseware, 
we  must  place  the  subject  into  its  proper  context.  None  of  our  representatives 
would  recommend  that  the  computer  be  utilized  as  the  sole  means  of  presenting 
instruction.  The  proper  mix  of  computer  managed  instruction,  and  computer 
assisted  instruction,  was  a controversial  topic.  Further,  experiments  into  such 
areas  as  multiple-track  strategies,  learner -control  courses,  and  confidence  testing, 
were  suggested.  The  need  for  experimentation  in  these  areas  is  readily  recognized; 
however,  this  would  cause  undue  burden  on  the  evaluation  of  the  current  system. 
Rather  than  dropping  the  ideas  we  are  currently  seeking  for  other  means  o'  testing 
these  innovative  ideas. 

It  was  in  tne  area  of  evaluation  that  the  majority  of  the  discussion  arose.  To  properly 
determine  where  we  should  go  in  the  future,  it  is  essential  that  the  evaluation  of  the 
current  project  be  accepted  by  all  concerned.  There  was  unanimous  agreement  that 
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the  plan  a s written  need  be  modified  to  take  into  consideration  the  shift  in  emphasis 
from  computer  assisted  instruction  to  computer  managed  instruction.  As  a result, 
this  office  has  already  embarked  upon  a major  project  in  cooperation  with  the  Army 
Research  Institike  and  the  Advanced  Research  Pi  jject  Agency  to  outline  a very 
definitive  plan  for  evaluating  not  only  Project  ABACUS  but  other  ongoing  projects 
in  the  field.  This  effort  represents  the  major  tangible  results  of  the  conference. 
Hopefully,  the  resulting  benefits  will  impact  not  only  on  the  military  but  on  academia 
and  in  the  industrial  training  world  as  well. 

In  conclusion,  I would  like  to  express  my  appreciation  to  the  representatives  of  the 
Army  Research  Institute  for  coordinating  the  conference;  to  ARPA  for  assisting  in 
the  sponsorship;  to  the  members  of  my  own  st  if*  'rom  the  Computerized  Training 
Systems  for  their  efforts  and  naturally  to  Fort  Gordon  and  the  US  Army  Signal  School 
for  acting  as  hosts. 


G.  B.  HOWARD 
COL,  SigC 

Product  Manager,  CTS 
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Biographical  Sketch 


O.  Dennis  Barnes  Ph.  D. 


Currently  Chairman  of  the  Department  of  Instructional 
Technology  at  the  Rochester  Institute  of  Technology.  Pre- 
viously Director  of  Curriculum  Development  and  Evaluation 
and  Director  of  Computer  Assisted  Instruction  for  the  National 
Technical  Institute  for  the  Deaf.  Holds  the  Ph.  D.in  Instruc- 
tional Technology  from  the  University  of  Southern  California. 
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A Training  Management  System 


bv 

O.  Dennis  Barnes,  Ph.  D. 

After  a site-visit  to  Fort  Gordon,  conversation  with  the  CTS  staff,  interaction 
with  the  project  civilian  and  military  consultants,  a review  of  the  literature  of  the 
CTS,  and  at  the  request  of  the  CTS  project  manager  this  paper  is  herewith  sub- 
mitted. The  first  section  is  concerned  with  observations  regarding  the  CTS  pro- 
ject and  the  second  (and  Primary)  section  is  concerned  with  a suggested  system 
for  managing  the  training  process. 

a.  CTS  Observations 

The  CTS  staff  seems  to  be  enthusiastic,  committed  to  the  project,  and  con- 
cerned that  the  project  will  succeed.  The  two  primary  areas  of  concern  are  the 
evaluation  design  and  the  generalizability  of  the  instructional  strategy. 

b.  Evaluation  Design 

(1)  Scriven*  is  attributed  with  the  notion  of  formative  vs.  summat.ve  evalu- 
ation. It  is  readily  apparent  that  members  of  the  CTS  staff  are  familiar  with  these 
notions  and,  at  least  in  the  Basic  Electronics  course,  have  applied  these  two  types 
of  evaluation.  It  is  suggested  that  in  the  evaluation  design  for  CTS  two  additional 
evaluation  components  be  well  specified:  student,  performance  and  program  per- 
formance . 

(2)  The  evaluation  of  "student  performance"  is  an  analysis  of  the  parameters 
associated  with  the  student's  acquisition  of  the  desired  behaviors.  These  para- 
meters include  length  of  time  for  acquisition,  degree  of  acquisition,  and  attitudes 
associated  with  acquisition.  The  evaluation  of  "program  performance"  is  an  an- 
alysis of  the  degree  to  which  the  instructional  program  elicited  the  desired  behavior. 

Parameters  associated  with  program  performance  are  efficiency  of  stimulus  frames, 
optimization  of  sequencing,  and  the  degree  to  which  skill  performance  is  attribut- 
able to  program  strategies.  Both  student  and  program  performance  should  be  ex- 
amined formatively  and  summatively. 

c.  Instructional  Strategies 

Although  CTS  has  an  instructional  model  it  tends  to  limit  the  way  in  which  a 
student  can  acquire  the  desired  behavior.  In  other  words,  for  any  given  set  of 
objectives,  there  is  only  one  instructional  technique  used  to  elicit  the  desired  be- 
havior. It  is  suggested  that  flexibility  be  designed  into  the  program  to  examine 
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various  strategies  associated  with  a set  of  objectives.  In  a CAl  context,  this  would 
mean,  for  example,  designing  a drill  and  practice  and,  perhaps  a tutorial  strategy 
for  a given  problem.  Data  could  be  collected  to  determine  the  general  nature  of  a 
strategy  which  works  for  a given  type  of  student.  The  work  in  Cognitive  Style  Map- 
ping has  applicability  here  in  terms  of  describing  student  learning  styles^. 

d.  Training  Defined 

(1)  The  most  intriguing  notions  to  me  to  result  from  the  recent  on-site  visit 
to  the  CTS  Project  were  the  statements  made  by  Colonel  Howard  and  summarized 
in  the  discussion  guide  for  the  training  working  group:  "The  current  thrust  of  Army 
thinking  and  planning  in  general  is  to  bring  about  significant  increments  in  training 
proficiency  for  a given  time  period  and/or  to  bring  about  a given  level  of  proficiency 
with  a significant  decrease  in  training"2 3. 

(2)  For  the  purposes  of  this  discussion,  the  word  training  will  bs  used  in  its 
usual  sense,  i.e. , the  process  by  which  a desired  behavior  is  elicited  from  an  indi- 
vidual or  a group  of  individuals.  The  product  of  training,  however,  will  herein  be 
defined  not  only  as  the  required  technical  behaviors,  but  also  communJcation  and 
social  behaviors.  Explicit  in  these  two  definitions  is  the  notion  that  training  can  be 
concerned  with  eliciting  from  either  an  individual  or  group,  communication,  social, 
and  technical  behaviors.  In  the  usual  sense,  these  behaviors  are  determined  by 
a careful  analysis  of  the  requirements  of  a job. 

(3)  Communication  behaviors  deal  with  effective  transmission  of  information. 
Social  behaviors  deal  with  successful  interactions  between  persons  or  machines  or 
vice-versa.  Technical  behaviors  combine  communications  and  social  behaviors 
and  deal  wiih  requirements  for  successful  completion  of  a job.  Technical  behaviors 
are  primarily  concerned  with  cognitive  and  psychmotor  behavior.4*  5 

e.  Establishing  Needs 

(1)  In  order  to  determine  jobs  which  require  training,  at  least  the  following 
parameters  should  be  examined:  1)  the  task  to  be  performed  by  a person  or  per- 
sons; 2)  the  interrelatedness  of  these  task,  with  other  job  tasks;  and  3)  the  need 
for  persons  to  perform  these  tasks  for  the  jhort  term  (1-3  years)  and  the  long 
term  (3-7  years). 

(2)  The  task?  co  be  performed  should  be  analyzed  by  describing  the  communi- 

cations, social,  and  technical  skills  required  for  successful  performance  of  the  job. 

At  least  one  way  of  examining  these  skills  is  to  determine  for  each  the  cognitive, 

affective,  and  psychomotor  skill  requirements.4*  5*  6 A determination  should 

also  be  made  regarding  the  number  of  persons  required  to  perform  the  tasks . If 
only  one  person  is  required  e.g.  typing  a letter,  the  training  process  need  only 
concern  itself  with  training  an  individual.  If  several  persons  are  required,  e.g. 
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repairing  a jet  engine,  then  a team  must  be  trained.  The  group  dynamic  or  social 
skills  will  be  much  more  complex  in  the  latter  than  in  the  former  case  and  will  re- 
quire iden;ification.  The  last  major  function  of  establishing  needs  is  the  determin- 
ation of  the  hierachy  of  skills  which  will  effect  the  desired  terminal  performance.2 * * * *  7 
This  hierarchy  should  describe  those  skills  which  are  prerequisite  to  the  successful 
performance  of  the  specified  set  of  tasks. 

(3)  Very  few  if  any  jobs  are  performed  in  isolation  vithout  the  influence  of 
the  products  of  other  jobs.  The  team  that  repair  a jet  engine  relies  on  other  iobs 
for  a host  of  results,  e.g.  data  regrrding  engine  performance,  repair  supplier 
generated  by  a supply  systems,  or  performance  specifications  provided  by  engineers. 
Those  jobs  on  which  a specified  job  is  dependent  should  be  identified  and  the  pro- 
ducts defined.  Without  this  information,  a job  for  which  a person  or  team  is  being 
trained  can  fail  because  of  a lack  of  information. 

(4)  Obviously  if  a particular  job  after  two  years  will  require  a 50  percent 
reduction  in  man-pov/er,  the  need  for  a training  program  should  be  seriously  ques- 
tioned. Perhaps  the  best  example  of  this  phenomena  occurs  with  a major  techno- 
logical innovation.  The  classic  example  would  be  the  shift  from  vacuum  tubes  to 
transistors.  Those  responsible  for  training  must  provide  themselves  with  a mech- 
anism for  determining  not  only  the  currency  of  the  job  skill  requirements  as  well  as 
the  number  of  workers  required. 

f.  On-the-Job  Training 

(1)  Stokes  says,  "Most  of  the  training  that  is  done  in  America  is  done  on  the 
job.  The  immediate  goal  of  on-the-job  training  is  to  develop  a skill  ....  Actually 
performing  the  work  is  the  only  way  a trainee  can  get  the  necessary  practice. . . . 

The  key  to  planning  (on-the-job)  training  is  the  analysis  of  the  job.  " 8,pp  54  & 55 
The  model  presented  below,  might  be  considered  an  on-the-job  model. 
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Train  - 


Perform - 


• Monitor 


^-Determine  Next  "Train' 


• Train 


(2)  In  this  model  the  job  to  be  performed  is  noted  by  the  appearance  of 

"perform".  Note  that  if  a hierarchy  of  skills  has  been  determined  the  individual 

or  team  will  only  perform  part  of  the  total  job.  The  total  set  of  job  related  tasks 

are  only  accomplished  after  a series  of  intermixed  training  and  performance  func- 
tions. Each  "train"  function  is  concerned  with  eliciting  only  certain  behaviors  of 

the  total  job;  not  the  entire  skill  set  of  the  job.  The  "perform"  function  should  not 

simply  be  practice  effects , although  they  closely  approximate  this . They  in  fact 


7 


should  be  integrated  with  actual  job  experiences.  This  is  possible  if  an  identified 
job  and  its  training  component  are  considered  as  an  integrated  system.  Consider 
for  example  secretarial  training.  It  seems  theoretically  feasible  for  a set  of  skills 
to  be  identified,  e.g.  filing,  typing,  answering  the  phone,  transcribing,  etc. 

Assume  that  the  hierarchy  pf  these  skills  would  progress  from  low  to  high  in  the 
following  way:  answering  the  phone,  filing,  typing,  and  transcribing.  The ’training" 
process  would  first  develop  in  an  individual  the  ability  to  answer  the  phone  properly. 
Conceivably  this  would  take  an  hour.  For  the  next  several  hours  the  individual 
would  answer  the  phone  on-the-job.  The  process  of  "monitoring"  would  determine 
the  acceptability  of  the  individual's  performance,  diagnose  strength  and  weaknesses, 
and  prescribe  remedial  training  or  logical  advancement. 

(3)  The  branch  from  "perform"  to  the  next  ’train"  is  obviously  linear  for  the 
trainee.  Each  new  training  function  for  each  trainee  will  follow  a sequential  and 
linear  pattern.  However,  from  an  overall  training  system  point  of  view,  the  se- 
quence process  is  non-linear.  This  becomes  clear  if  a "monitor”  function  is  de- 
fined by  the  following  sub -processes:  1)  observe  performance  and  2)  determine 
level  of  acceptability. 

(4)  The  level  of  acceptability  will  have  been  determined  as  an  integral  com- 
ponent of  the  behavior  specification.  The  "determine  next  train"  functional  decisions 
process  will  have  been  specified  as  integral  part  of  the  skill  hierarchy  determination. 

(5)  Given  this  model,  it  can  be  seen  that  training  is  a continuous  activity 
conducted  in  concert  with  the  requirements  and  expectations  of  the  jobs.  Training 
as  suggested  here  is  not  considered  a discrete  activity. 

(6)  Given  this  proposition,  it  is  not  difficult  to  see  that  the  "train"  function 
diminishes  as  the  "monitor"  function  indicates  acceptable  "performance".  There- 
fore, the  two  functions  which  remain  ongoing  and  dynamic,  are  the  performance 
and  monitor  functions . 

g.  A Training  Management  System 

(1)  A training  system  as  described  above  can  only  perform  as  well  as  its 
management  system  performs.  At  least  two  premises  suggested  above  should  be 
considered  major:  1)  the  definition  of  job-related  skills  in  terms  of  communica- 
tion, social,  and  technical  skills  and  2)  the  integration  of  training  and  performance. 
The  latter  point  is  possible  only  if  the  duration  of  trainee  or  teams  productivity  and 
training  are  integrated  into  one  major  system. 

(2)  A job  and  training  for  a job  can  be  considered  an  interrelated  process. 

This  is  feasible  if  a trainee's  or  team  of  trainees'  performance  is  systematically 
examined  by  means  of  a well  designed  evaluation  procedure.  The  training  process 
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would  be  discontinued  when  the  overall  level  of  acceptability  is  achieved.  Similarly, 
it  would  be  initiated  when  an  upward  or  horizontal  move  is  deemed  appropriate . 

Such  a training  process  would  be  feasible  given  the  organizational  structure  on 
page  10. 

(3)  The  research  section  would  be  responsible  for  identifying  and  answering 
problems  specifically  related  to  the  efficiency  of  the  training  process.  This  group 
would  be  an  autonomous  group  reporting  to  the  manager.  The  job  analysis  section 
would  be  responsible  for  determining  the  communications,  social,  and  technical 
skills  requirements  of  the  jobs;  determining  the  short  and  long  term  worker  require- 
ments; determining  the  job  skill  hierarchy;  and  determining  the  interrelatedness  of 

a job  with  other  jobs . 

(4)  The  personnel  management  system  would  be  responsible  for  ensuring 
the  proper  selection  and  placement  of  trainees.  This  section  would  obtain  concrete 
data  from  the  training  division  evaluation  section  in  determining  career  paths  and 
job  performance  for  each  of  the  po-sonnel  in  its  control. 

(5)  The  training  division  would  have  overall  responsibility  for  the  develop- 
ment of  the  training  system.  It  would  be  responsible  for  training  those  trainees 
and  teams  which  have  been  selected  by  the  personnel  management  system.  The 
training  division  is  not  limited  to  training  in  a discrete  period  of  time,  ft  is  re- 
sponsible for  training  over  a continuous  period  of  time  depending  upon  the  nature 
of  the  training  task  and  the  amount  and  the  level  of  performance  obtained  from  the 
trainees. 

(6)  The  training  division  is  divided  into  three  sections:  evaluation,  training 
system  technologists,  and  materials  producers.  The  evaluation  section  is  respon- 
sible for  examining  on  a unit  by  unit  basis  the  performance  of  the  trainees;  de- 
termining, according  to  a predetermined  sequence  algorithm,  the  sequence  that 
individual  and  team  trainees  will  proceed  through  the  training  program,  and  ex- 
amining the  performance  of  the  training  program  itself,  e.g.  the  effectiveness 

of  materials,  and  the  efficiency  of  administrative  and  monitoring  systems. 

(7)  The  training  systems  technologists  section  is  composed  of  what  might 
commonly  be  considered  instructional  developers.  These  individuals  are  responsible 
for  validating  needs  determined  by  the  job  analysis  section,  specifying  the  terminal 
performance  objectives  expected  of  the  trainees,  determining  appropriate  strage- 
gies,  and  implementing  and  monitoring  the  training  system.  The  materials  pro- 
ducers are  concerned  with  producing  audiovisual  and  instructional  materials. 

This  section  would  include  photographers  as  well  as  computer  assisted  instruction 
specialists. 

(8)  The  fourth  and  final  component  of  the  training  management  system  would 
be  that  of  the  job  co-ordinator.  This  section  is  the  team  which  manages  the  actual 
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job  performance  requirements.  In  the  secretarial  example  stated  earlier,  the 
function  co-ordinator  might  be  the  supervisor  of  a secretarial  pool. 

(9)  This  management  system  suggests,  in  part,  that  training  should  not  be 
considered  a discrete  activity  but  as  a continuous  activity  based  on  a hierarchy  of 
skills  and  the  ability  of  the  trainees  to  perform  as  determined  by  the  monitoring 
function. 

(10)  One  way  of  examining  the  effectiveness  of  such  a management  system 
would  be  to  identify  one  job  area  in  which  the  Army  has  extensive  training  experi- 
ence. This  job  area  would  then  be  reorganized  in  such  a way  that  the  trainees 
would  be  integrated  with  the  actual  job  requirements. 

h.  Summary 

The  assumption  made  for  the  discussion  presented  above  is  that  the  responsibility 
of  training  is  to  modify  the  behavior  of  individuals  who  are  expected  to  perform  on 
jobs  at  acceptable  levels  of  performance.  Jobs  are  defined  in  terms  of  tasks. 

Tasks  in  turn  are  defined  as  an  identifiable  set  of  communication,  social,  and  tech- 
nical skills.  These  skills,  it  is  assumed,  can  be  analyzed  and  ordered  into  a 
hierarchy  of  skills.  The  lowest  on  the  hierarchy  being  required  to  be  obtained  in 
order  for  successful  acquisition  of  the  highest  skill  in  the  hierarchy.  In  order  to 
modify  a trainee  or  team  of  trainees  behavior  in  an  efficient  manner  it  is  suggested 
that  the  function  of  training  be  interrelated  with  the  actual  performance  of  newly 
acquired  skills.  In  conjunction  with  the  performance  of  these  skills  a monitoring 
function  occurs  which  determines  the  acceptability  of  the  performance.  In  the  case 
where  the  performance  is  not  acceptable  the  individual  is  sequenced  to  either  re- 
medial training  procedures  or  recycled  through  previously  experienced  training 
procedures.  In  the  event  that  the  performance  is  acceptable  the  individual  is 
appropriately  sequenced  to  the  next  training  module.  The  total  set  of  training 
performance  and  monitoring  functions  need  not  be  in  a sequential  manner.  They 
can,  in  fact,  be  parallel.  In  order  to  effect  such  a training  pattern,  it  is  neces- 
sary to  manage  job  performance  and  training  expectations  in  such  a way  as  to 
efficiently  and  effectively  obtain  the  desired  behavior.  An  organizational  system 
is  suggested  in  which  four  major  components  are  considered:  job  analysis  section, 
personnel  management  system,  training  division,  and  job  co-ordinator.  In  addition 
to  these  four  sections,  a research  arm  which  is  quasi  autonomous  examines  funda- 
mental research  questions  regarding  the  generalizability  of  the  training  areas.  A 
specific  suggestion  is  made  that  a carefully  selected  job  area  be  used  to  determine 
the  effect  of  such  a system. 
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Computerized  Training  Systems 


Consultant  Conference  Report 


Jan  Danford 


Intoduction 


a.  This  report  will  deal  with  essentially  three  areas  and  will  address  them  in  the 
following  order:  CMI-CAI  and  Task  Analysis,  student -trainee  needs  assessment, 
and  formatting  of  instructional  packages.  While  it  is  virtually  impossible  to  speak 
to  any  one  area  without  touching  on  the  others,  or  keep  from  overlapping  into  areas 
that  cannot  be  dealt  with  in  full  in  this  paper,  the  attempt  will  be  made  10  handle 
each  separately.  The  order  of  presentation  is  also  vital  in  that  one  is  the  outgrowth 
of  the  other  and  the  quality  and  completeness  is  dependent  upon  the  kind  of  job  done 
at  every  step. 


b.  The  writer  chose  to  address  these  areas  because  of  a background  both  in  the 
developmental  aspects  of  each  as  well  as  two  and  one-half  years  of  implementation. 
The  direction  and  recommendations  made  are  based  on  what  was  seen,  heard,  and 
read  and  are  presented  accordingly. 


CMI/CAI — Task  Analysis 


a.  It  was  expressed  verbally  and  is  obvious  from  the  state  of  development  and 
the  prevailing  atmosphere  that  ABACUS  is  torn  as  to  the  direction  it  should  be 
taking.  To  date  they  have  dealt  with  CAI  but  have  recently  seen  a need  for  CMI. 

I totally  support  this  position  and  further  feel  that  CAI  should  be  thought  of  as  only 
one  possible  instructional  tool  in  an  entire  systems  rpproach  to  instruction.  ABACUS 
has,  or  is,  backing  into  CMI.  Since  the  same  kind  of  analysis  and  preparation  that 
is  used  in  CAI  is  necessary  for  CMI,  the  change  in  direction  will  not  be  that  drastic. 


b.  However,  from  what  was  observed,  the  writer  feels  that  the  quality,  depth, 
and  emphasis  will  change. 


c.  Several  individuals  at  Fort  Gordon  were  talked  with  in  regard  to  what  was 
termed  a task  analysis.  It  was  defined  as  an  identification  of  skills  from  the 
manuals;  manuals  written  by  the  manufacturer  of  a given  product.  I would  like 
to  suggest  a different  system  for  developing  a task  analysis  and  a new  format. 
While  the  task  analysis  format  and  system  that  is  presented  here  is  probably  not 
absolutely  fool  proof,  it  has  done  what  it  was  designed  to  do  for  fifty  occupations  in 
Michigan,  and  I feel  certain  it  will  do  the  same  for  ABACUS.  Again,  I must  stress 


Preceding  page  blank 


i 


15 


the  need  for  a very  thorough  task  analysis.  Without  it  the  content  of  the  instructional 
packages  (modules)  are  at  the  mercy  of  a technical  writer.  The  task  analysis  also 
lays  the  foundation  for  the  identification  of  the  components  of  the  "needs  assessment.  " 

d.  In  order  to  do  quality  individualizing  of  instruction  that  addresses  the  specific 
job  skills  needed  for  a particular  occupation,  it  is  necessary  to  identify  precisely 
what  it  is  that  a person  does  to  function  effectively  and  satisfactorily  as  defined  by 
the  fact  he  has  retained  the  job  and  received  satisfactory  ratings . You  have  to  go  to 
the  person  doing  the  job  in  the  field.  You  go  to  him  with  an  instrument  he  can  re- 
spond to.  Let's  take  a close  look  at  a task  analysis  system  and  format  that  does 
just  that. 

Task  Analysis 

a.  Given  the  assumption  that  the  occupation  selected  is  one  in  which  pretraining 
is  needed  and  that  jobs  do  exist  and  are  projected  to  exist  for  a reasonable  time, 
individuals  who  are  functioning  in  the  said  occupation  are  brought  in  to  draft  the 
tasks  that  will  be  listed  in  a precise  order  on  a prescribed  format.  These  indivi- 
duals are  referred  to  as  "Advisory  Channels"  and  it  is  suggested  that  a minimum 

of  five  to  seven  are  used.  Depending  upon  the  nature  of  the  field,  people  repre- 
senting different  size  units  or  different  circumstances,  but  still  carrying  the  same 
job  classification,  must  be  included.  It  has  been  found  that  inputs  from  this  group 
are  better  and  faster  if  they  are  never  brought  together  as  a committee  but  work 
independently  funneling  their  data  into  a central  leader.  The  leader's  task  is  to 
sift  the  information  into  one  document. 

b.  The  Advisory  Channels  are  asked  to  provide  data  in  five  categories.  They 

are  asked  to  do  so  in  a preorganized  manner.  The  first  area  is  to  list  all  Job 
Skills  that  a person  does  on  their  job.  These  are  explained  as  the  hands-on,  ob- 
servable functions  that  anyone  could  record  as  visable,  if  someone  followed  him 
around.  These  skills  are  grouped  under  headings  (units  of  learning).  This  assists 
the  advisor  in  concentrating  on  all  skills  (tasks)  under  that  heading.  This  also 
helps  the  person  who  will  respond  to  the  document  in  ascertaining  its  completeness 
and  continuity.  The  advisor  starts  all  skill  statements  in  the  same  way:  The 
technician  is  able  to . 

3.  The  second  category  speaks  to  the  Knowledges  needed  to  do  the  list  of  skills. 
Again,  they  are  grouped  under  the  same  headings  and  must  be  tied  to  a skill  or 
skills.  This  is  done  in  order  to  keep  the  "excess  baggage"  out  of  the  resulting 
curricula.  Knowledges  are  defined  as  an  understanding  needed  in  order  to  problem 
solve.  They  are  not  just  simple  pieces  of  information,  but  rather  information  in 
order  to  "decide".  The  third  category  deals  with  Behaviors  peculiar  to  the  job. 

Items  dealing  with  ability  to  take  responsibility,  make  decisions,  handling  of 
authority,  etc.  fall  under  this  category.  The  fourth  category  deals  with  Basic 
Skills . Basic  Skills  are  defined  as  the  reading  and  arithmetic  skills  necessary  to 
function  on  the  job.  Finally,  the  Iasi  category  deals  with  the  Pieces  of  Equipment 
and  Tools  that  must  be  mastered. 
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d.  You  will  find  attached  a copy  of  the  format  for  the  first  two  categories.  You 
will  notice  that  Exhibit  I,  Job  Skills,  asks  the  interviewer  to  indicate  entry  and 
frequency  of  performance.  Both  are  defined  and  parameters  set.  Of  course  these 
can  be  changed  if  needed.  Perhaps  another  breakout  might  be  needed  indicating 
battle  and  nonbattle  conditions  that  will  effect  both  entry  time  and  frequency  of 
performance.  Exhibit  n deals  with  the  accompanying  job  knowledges  and  sorts  out 
the  "nice  to  knows"  from  the  "absolutes.  " Since  the  relation  between  skills  and 
knowledges  has  been  emphasized  during  the  writing  of  the  instrument,  this  will 
further  help  in  developing  a curriculum  hierarchy.  The  tying  of  knowledges  to  skills 
will  assist  in  insuring  that  instructional  packages  will  present  the  theory  and  then 
reinforce  it  by  having  the  student -trainee  immediately  apply  it.  We  thus  do  away 
with  theory  modules  and  skill  modules  and  end  up  with  an  integrated  (skill-knowledge) 
module. 

e.  Taking  just  a moment  to  back  up  and  deal  with  the  term  curriculum  hierarchy, 

I think  a further  explanation  is  needed.  This  refers  to  the  fact  that  the  absolutes  are 
required  and  the  "nice  to  knows  "can  be  taken  if  the  student  has  time  available  to 
him. 


f.  Both  categories  number  each  task  separately  for  use  by  the  computer.  Let's 
take  a look  at  exactly  what  has  to  be  done  in  order  for  the  computer  to  analyze  the 
resulting  data.  If  you  are  dealing  with  several  occupations  within  the  same  occu- 
pational family,  you  still  use  one  instrument  and  one  leader,  but  separate  Advisory 
Channel  groups.  The  leader  sorts  the  common  tasks  being  sure  where  they  are 
identical  they  are  worded  accordingly.  Once  the  instrument  is  finalized,  only  the 
occupation  title  changes . (See  Exhibit  I. ) Directions  are  given  to  each  person 
being  interviewed  to  respond  only  to  the  items  that  pertain  to  him,  the  occupation 
at  the  top  cf  the  analysis  form.  However,  if  you  are  just  going  to  offer  one  occu- 
pation within  the  family,  an  analysis  need  be  done  just  on  that  one.  The  first  case 
does  allow  you  to  establish  a career  ladder  by  defining  the  commonalities  between 
the  occupations.  This  enhances  the  learning  environment  as  each  student  finds  his 
own  level.  He  goes  as  far  as  he  can  as  fast  as  he  can  along  that  ladder,  mastering 
each  skill  to  the  performance  criteria  indicated  until  he  can  do  no  more. 

g.  I have  enclosed  copies  of  computer  printouts  (Exhibits  HI,  IV,  V)  for  Job 
Skills,  Commonalities,  and  the  relationship  of  Job  Skills  to  Job  Knowledges. 

h.  Now  let's  take  a look  at  the  administering  of  the  survey  instrument.  The 
size  of  your  population  should  be  determined  and  categorized.  Once  it  is  stratified 
then  a percentage  should  be  randomly  selected  from  each  category.  Our  population 
sample  in  Michigan  had  to  be  influenced  by  time  and  dollars.  We  did  add  a few 
rules  that  helped  insure  the  validity  of  the  data  received  and  that  was:  (1)  the 
person  in  said  job  classification  must  have  held  the  job  for  a minimum  cf  one  year, 
but  not  more  than  five,  and  (2)  they  had  to  be  holding  the  job  at  present,  no  super- 
visors or  foremen  could  respond.  The  reasons,  of  course,  in  the  first  case  is 
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simply  that  one  year  allowed  the  entry  items  to  be  distinguished  from  the  nonentry 
and  the  maximum  of  five  years  helped  prevent  a memory  lag  of  what  was  and  wasn't 
entry.  The  second  case  is  supported  by  research  in  that  supervisors,  etc. , and 
the  worker,  both  filling  out  the  same  survey,  find  there  is  as  much  as  a thirty-five 
percent  discrepancy  in  the  responses. 

i.  Once  the  instruments  are  filled  out,  the  computer  does  the  rest.  It  accepts 
a task  that  receives  a given  number  of  responses  in  entry  and  frequency,  and  sends 
into  review  those  that  do  not.  Itspells  out  the  commonalities.  Inctherwords.it 
gives  you  a very  good  base  from  which  to  develop  curriculum  systems  and  materials. 

Instructional  Packages  (Modules) 

a.  If  your  first  phase,  task  analyses,  has  been  well  done,  the  task  statements  are 
the  performance  you  desire  and  thus  the  performance  part  of  the  performance  objec- 
tive. You  have  a very  detailed  job  description  or  course  outline  if  you  prefer.  Each 
statement  can  now  be  expanded  into  a module  of  instruction  allowing  for  CMT  and 
appropriate  CAI. 

In  addressing  the  area  of  modules,  it  will  be  presented  in  the  following  categories: 

(1)  Format 

(2)  Comprehension 

(3)  Types 

(4)  Testing 
Format 

a.  The  adoption  of  one  format  assures  uniformity  and  eases  the  student's  move- 
ment both  through  a course  and  from  course  to  course.  There  will  be  those  indivi- 
duals who  will  want  to  develop  their  own  way  or  modify  the  accepted  format,  however, 
we  have  found  that  unless  the  modification  is  desirable  for  the  instructional  system 
as  a whole,  it  should  not  be  allowed.  We  have  developed  and  are  using  one  format 
that  works  equally  well  for  all  of  our  seventeen  (17)  courses  and  fifty-four  (54) 
occupations.  The  Army  may  feel  that  this  format  will  work,  or  a reordering  up 
of  the  same  information  is  necessary,  whatever  the  point  is  to  gain  consistency. 


b.  There  are 
tification  (cover) 
standing  needed, 
lary  list.  Let  us 


five  parts  to  a module  whether  it  is  used  for  CMI  or  CAI:  (1)  iden- 
page,  (2)  the  performance  expected,  (3)  the  information  and  under- 
(4)  the  actual  steps  in  learning,  and  (5)  the  technical  work  vocabu- 
discuss  them  in  the  order  just  listed.  This  is  the  sequence  that 
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through  trial,  error,  and  revision,  has  worked.  A copy  of  each  follows  with 
appropriate  explanations. 


c.  Page  23  is  the  identification  page  and  thus  serves  as  a cover  sheet.  If  this 
module  were  on  a slide  (slide-tape  presentation),  it  would  serve  the  same  purpose. 
The  reader  should  realize  that  for  ease  of  presentation,  the  written  module  is 
being  used  as  an  example,  but  the  same  would  apply  no  matter  what  medium  was 
selected.  Most  of  it  is  self  explanatory,  but  level  and  occupation  should  be  ex- 
panded. Level  refers  to  the  type  of  module  (skill,  transfer,  etc.)  and  occupation 
demands  that  all  occupations  that  require  the  mastery  of  this  module  be  listed. 

This  gives  the  students  a chance  to  see  commonalities  between  occupations  and 
courses. 

d.  We  struggled  with  many  performance  objective  formats  and  have  found 

this  to  be  the  most  readable  and  understandable  for  the  greatest  majority  of  the 
students.lt  still  encompasses  the  essential  ingredients  of  conditions,  perform- 
ance and  criteria.  (See  page  24  . ) 

e.  Many  times  there  is  information  needed  by  the  students  before  they  can 
start  the  actual  steps  in  learning  a skill.  This  is  where  simple  information  is 
given,  the  reason  it  is  necessary,  and  the  introduction  of  knowledges  needed  to 
do  or  solve  the  succeeding  steps.  This  is  where  the  relationship  of  the  Job  Skills 
and  Job  Knowledges  becomes  evident  and  the  instructor  presents  it  here  and  in 
the  ensuing  steps. 

f.  The  Job  Step  Section  on  page  26  is  a very  detailed,  programmed  step  by 
step  process  of  learning  the  identical  task.  It  should  be  presented  in  a very  re- 
laxed speaking  narrative  tying  the  knowledges  to  each  step  as  they  apply,  all  the 
while  reminding  the  student  of  pertinent  past  experiments  and  applicable  learnings . 
It  should  be  so  clear  it  would  be  as  if  you  were  verbally  explaining  the  task  to  a 
very  slow  learner. 

g.  Finally,  new  words,  especially  technical  words,  will  be  introduced  to  the 
student.  (Page  28  ) The  beginning  modules  will  be  heavy  in  this  area  with  it 
falling  off  toward  the  end.  This  should  be  used  as  a minidictionary,  keeping  the 
definitions  as  simple  as  possible. 

h.  In  writing  modules  you  should  start  with  the  very  simple,  moving  to  the 
more  complex,  building  upon  previously  gained  skills  and  knowledges.  By  the 
same  token,  you  must  start  with  the  concrete  easily  demonstrable  skills,  to 

the  more  abstract  that  might  require  a student  to  problem  solve  and  troubleshoot. 
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Every  module  should  have  to  go  through  and  pass  several  tests.  First,  the 
reading  level  of  all  modules  should  be  constant . A fourth  or  fifth  grade  reading 
level  is  suggested,  and  writers  can  be  taught  to  do  this  with  comparative  ease. 
Second,  an  individual  totally  unfamiliar  with  the  subject  matter  should  silently 
read  it  for  continuity  and  flow.  They  should  then  answer  one  question  without 
referring  back.  What  is  it  you  should  be  able  to  do  when  you  finish  the  mod? 

This,  another  individual  should  read  it  aloud  to  someone  else.  This  will  point 
out  glaringly,  run  on  sentences,  grammatical  errors,  inconsistencies,  and 
punctuation  mistakes.  Fourth,  a brand  new  student  whose  reading  level  tests 
out  at  about  fourth  grade  should  do  the  module.  An  evaluator,  not  the  module 
writer,  should  be  with  the  student  to  check  for  stumbling  blocks,  assumption  of 
knowledge,  etc.  While  this  sounds  involved  and  somewhat  time  consuming,  it  does 
not  take  that  much  time  and  should  improve  the  end  product  tremendously. 


ss  and  Testing 


a.  While  at  Fort  Gordon,  we  had  an  opportunity  to  get  a look  at  some  of  the 
written  materials  being  used  by  the  students.  It  was  apparent  that  a great  per- 
centage was  at  a high  level  of  learning,  in  other  words  troubleshooting  (application 
of  theory).  There  was  some  written  at  a level  just  below  troubleshooting,  the 
synthesis  or  project  level.  It  did  not  appear  as  if  the  curriculum  had  gone  from 
the  simple  to  the  complex,  and  concrete  to  the  abstract,  but  had  instead  jumped 

in  somewhere  in  the  middle.  It  is  therefore  recommended  that  there  be  at  least 
four  levels.  Each  level  will  build  upon  the  other,  reinforcing  the  learning  through 
problem  solving  and  practice. 

b.  The  first  level  should  be  the  single  skill  or  operations  module.  Often  mock- 
up  trainer  units,  workbooks,  simulators,  etc.  are  used  to  teach  the  single  skill . 

It  is  a straight  forward  step-by-step  programmed  learning  process  that  is  set 
forth  in  the  most  simplistic  of  manner. 


c.  The  second  level  builds  upon  the  first.  Inis  module  is  called  a synthesis, 
integrated,  or  project  module.  The  term  is  not  particularly  important,  but  the 
concept  is  vital.  The  skills  are  brought  together  into  one  module  or  a related 
series  of  modules  dealing  with  one  concept.  This  allows  the  student  to  see  the 
relationship  of  the  part  to  the  whole. 


d.  The  third  level  deals  with  troubleshooting  or  problem  solving  and  thus  is  a 
level  that  might  not  be  mastered  by  all  students.  A situation  including  its  para- 
meters is  set  before  the  student  with  only  guidelines  for  his  thought  processes, 
not  specific  how  to  steps . The  steps  taken  must  be  spelled  out  by  the  learner  to 
be  checked  by  the  instructor.  This  encourages  understanding  as  opposed  to  me- 
chanically learning  a task. 

e.  Level  four  is  a transfer  module.  Since  any  training  environment  has  its 
limitations,  a student  must  be  prepared  to  handle  the  unfamiliar.  A student  may 
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learn  to  do  a process  a certain  way  or  to  handle  a certain  brand  of  equipment  and 
run  into  a different  piece  of  equipment  or  way  of  doing  a task . He  must  be  able  to 
transfer  his  learning  from  a familiar  to  a similiar,  but  not  identical  situation. 

f . The  testing  materials  that  were  available  for  review  appeared  to  deal  with  the 
cognitive  areas.  It  is  suggested  that  with  the  developing  of  modules  as  previously 
laid  out,  that  the  demonstration  of  the  skill  will  indicate  that  theory  has  been 
mastered  and  thus  applied.  Cognitive  testing  under  this  system,  would  be  just  to 
to  check  point  along  the  instructional  path  and  not  the  end.  As  a matter  of  fact. 
Level  HI  can  act  as  a pretest  for  purposes  of  determining  individual  curriculum 
starting  points.  This  will  give  you  a good  measure  of  both  cognitive  and  psycho- 
motor learning. 

Needs  Assessment 

a.  Perhaps  one  of  the  most  frustrating  things  for  a learner  is  to  find  he  is  ill- 
equipped  to  handle  the  material.  A good  Needs  Assessment  will  allow  each  learner 
to  start  where  he  should,  giving  him  credit  for  competencies  previously  learned 
and  prescribing  training  in  areas  where  he  needs  it.  There  are  three  levels  of 
Needs  Assessment  depending  upon  how  academically  handicapped  your  students/ 
trainees  may  be. 

b.  The  first  should  be  given  to  all,  and  it  deals  with  the  equated  arithmetic, 
reading,  and  behaviorial  skills  needed  in  order  to  be  tained  in  the  identified  occu- 
pation. This  will  allow  the  instructional  system  to  provide  pretraining  basic  skills. 
Most  students  have  virtually  no  problems  handling  this  as  they  are  being  asked  to 
do  only  what  is  necessary  to  succeed.  The  logic  is  obvious  as  you  can  show  them 
where  they  are  at  present  and  where  they  need  to  go.  In  a few  students  the  de- 
ficiency will  be  so  insignificant  that  often  the  basic  skill  and  occupational  training 
can  be  done  concurrently. 

c.  The  second  level  of  Needs  Assessment  should  be  done  where  there  is  a 
doubt  as  to  the  individuals  ability  to  handle  the  material,  even  if  the  above  basic 
skill  training  is  provided.  This  is  usually  done  with  the  academically  damaged 
student.  In  every  skill  to  be  learned,  there  is  an  assumption  of  past  learning  that 
goes  beyond  the  reading  and  arithmetic . These  are  the  things  we  would  take  very 
much  for  granted.  Case  in  point,  the  trainee  is  being  taught  to  take  a pulse.  The 
assumption  might  be  that  the  student  knows  what  a pulse  is  and  what  it  represents . 
Another  might  be  to  check  a starter  on  a car.  One  of  the  assumptions  might  be 
that  he  even  knows  how  to  start  a car  or  where  the  starter  is  located. 


d.  Any  training  situation  includes  assumptions  of  prior  learning,  a few  can  be 
taken  care  of  by  including  them  in  the  modules,  other  should  be  pretested. 
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e.  The  third  and  least  used  level  will  be  reserved  for  special  cases.  Every 
occupation  has  physical  activities  and  conditions  that  may  preclude  some  people 
from  being  trained.  These  will  measure  a person's  dexterity,  coordination,  ability 
to  discriminate,  ability  to  remember,  and  follow  instructions  both  written  and  ver- 
bal. The  assessment  will  deal,  to  use  an  example,  with  Bi-Manual  Coordination, 
Eye-Hand-Foot  Coordination,  Eye-Hand  Coordination,  Work  Rhythm,  and  so  on.  ’ 
Further,  if  you  needed  to  know  whether  a person  could  be  trained  for  let's  say  a 
job  of  Rubber  Stamper.  You  would  check  him  on  the  following  activities  by  means 
of  a prescribed  work  sample:  Neatness,  following  verbal  directions,  following  a 
model,  counting  ability,  eye-hand  coordination,  and  work  rhythm.  If  you  were 
considering  training  him  as  a Sign  Maker,  you  would  check  spatial  discrimination, 
form  discrimination,  planning  ability,  neatness,  organizational  ability,  eye-hand 
coordination,  and  measuring  ability. 

Summary 

The  three  areas  discussed  are  components  essential  in  the  establishment  of 
an  individualized  instructional  system.  A firm  base  is  necessary,  and  the  writer 
feels  strongly  that  CMI  should  be  developed  prior  to  CAI.  Then,  CAl  can  take  its 
rightful  place,  as  but  another  instructional  tool. 


# 

DATE: 

LEVEL: 
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Industry  has  told  us  that  they  require  you  to  know  and  be  able  to  do  the 
following  skills . Take  your  time.  Read  very  carefully.  Ask  questions . 

Read  the  objective  as  if  it  were  one  sentence — your  instructor  will  show  you  how. 

PERFORMANCE  OBJECTIVE  I 

YOU  ARE  GOING  TO  LEARN: 


I 


AND  IN  ORDER  TO  DO  THIS  YOU  WILL  BE  GIVEN: 


WE  WILL  KNOW  YOU  CAN  DO  IT  (PASS)  WHEN: 

TEST 

AND 

EVALUATION 


TURN  TO  NEXT  PAGE 
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In  order  to  be  able  to  understand  and  do  the  job  steps,  do  everything 
on  this  page  before  going  to  the  next  page.  Ask  your  instructor  to  help 
if  you  need  it. 


— 

MATERIALS 

WHERE 

INFORMATION 

(List  chapter, 

YOU  WILL 

page,  etc.) 

FIND 

MATERIAL 

-4- 


JOB  STEPS 


DIRECTIONS:  Do  in  the  exact  order  listed. 


Step  1.  For  the  materials  and  equipment  you  will  need 
to  complete  the  following  steps,  reread  the 
Goal  (page  two) . Any  additional  material 
needed  to  learn  the  skills  will  be  listed  in 
the  Equipment  G Materials  column  on  this  page. 


Equipment  G Materials 
Needed  (See  Page  2) 


Turn  to  Next  Page 
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Equipment  & Materials 
Needed  (See  Page  2) 


JOB  STEPS 


DIRECTIONS:  Do  in  the  exact  order  listed. 


Now  have  your  instructor  check  your  work 
(See  page  2 for  Test  4 Evaluation.) 


! 


NODULE 


TECHNICAL  WORDS 


Found  In  Job 
Stop  Nunber 


Words 


Meaning 


Exhibit  I.  CRITICAL  INCIDENT  ANALYSIS 


•It  was  required  of  you  within  the  first  60  days  of  the  job. 


Exhibit  II.  CRITICAL  INCIDENT  ANALYSIS 


Exhibit  in.  Job  Skills 


— 86%/13  0fi%/01  60%/09  00%/00  13%/02  00%/00  00%/00  Accept 


133  13044  Computer  Operator 

132  13044  Unit  Record  Operator 


Exhibit  IV.  Commonalities  Characteristic  Report  (Cont) 


12/15/71 


Exhibit  V.  Job  Knowledges  Related  to  Job 


13000  Data  Processing  Occupations 


Job  Knowledge 


Job  Skill 

13001 

13002 

13003 

13004 

13005 

13006 

13007 

13008 

13009 

13010 

13011 

13012 

13013 

13014 

13015 

13016 

13017 

13018 

13019 

13020 

13021 

13022 

13023 

13024 

13025 

13026 

13027 

13028 

13029 

13030 

13031 

13032 

13033 

13034 

13035 

13036 

13037 

13038 

13039 

13040 

13041 

13042 

13043 

13044 
130  <5 

13046 

13047 

13048 

13049 


J 


I 


02/03/72 


♦♦  Cominnalities  Characteristic  Report  ♦* 


Page  002 


j 
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Characteristics  - To  be  able  to  perform  routine  maintenance  on  machines. 


♦Required  in  Programs 


OCC  Title-Name 

13000  Data  Processing  Occupations 

132  13059  Unit  Record  Operator 

133  13059  Computer  Operator 


Characteristics  - To  block, sort  and  select  certain  fields  from  the  card. 


♦Required  in  Programs 

OCC  Title -Name 

13000  Data  Processing  Occupations 

133  13023  Computer  Operator 

132  13023  Unit  Record  Operator 

Characteristics  - To  clear  card  jams  from  all  of  the  machines. 

♦Required  in  Programs 

OCC  Title -Name 

13000  Data  Processing  Occupations 

133  13058  Computer  Operator 

132  13058  Unit  Record  Operator 


Characteristics  - To  clear  the  machine  of  card  jams  if  it  occurs. 

♦Required  in  Programs 

OCC  Title -Name 

13000  Data  Processing  Occupations 

133  13044  Computer  Operator 

132  13044  Unit  Record  Operator 

Characteristics  - To  code  for  programming:  prepare  mathematical  statement 
of  common  business  problems  to  facilitate  programming 
procedure . 

♦Required  in  Programs 
OCC 

13000  Data  Processing  Occupations 

133  13072  Computer  Operator 

134  13072  Programmer 


Biographical  Sketch 


William  F.  Fitzgerald,  Ph.  D. 
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COST/BENEFIT  ANALYSIS 
Project  ABACUS 


Submitted  in  partial  fulfillment  of  Task  Order  75-129  by: 

William  F.  Fitzgerald,  Ph.  D. 

Director,  Division  of  Educational 
Resources  and  Studies 
American  Association  of  Dental  Schools 
February  1975 


Section  1.  Cost/Benefit  Analysis 

a.  General  Purposes 

Cost  analysis  is  a complex,  multi-faceted  process  with  diverse  purposes. 

At  a minimum,  it  is  a procedure  of  categorizing  and  quantifying  an  activity  in  the 
common  metric  of  money.  For  the  purposes  of  the  ABACUS  Project,  the  following 
objectives  of  cost  analysis  will  be  identified,  and  some  of  them  will  be  discussed 
in  detail. 

(1)  Activity  Separation.  In  a complex  environment  it  is  a desirable  objective 
to  determine  the  costs  associated  with  each  sub-activity.  The  presumption  of  this 
purpose  is  that  if  priorities  or  resources  change,  this  analysis  will  provide  admini- 
strators with  the  data  that  are  needed  to  respond  to  change. 

(2)  Staff  Efficiency.  In  computer  assisted  instruction,  there  are  numerous 
tasks  to  be  accomplished.  Tersely  stated,  entry  level  behaviors  (achievement) 
and  learner  characteristics  (aptitude)  must  be  specified;  content  must  be  identified 
and  organized;  and  instructional  program  (assuming  that  systems  software  and  an 
applications  language  exist)  must  be  written  and  coded;  code  must  be  entered  into 

a computer  in  machine -readable  form;  material  must  be  evaluated  on  a representa- 
tive sample  of  the  target  population;  and  revisions  in  content  and  logic  must  be  made 
to  reflect  the  results  of  the  field  test.  For  a single  person  to  perform  all  of  these 
tasks,  training/education  in  the  skills  not  possessed  must  be  accomplished,  and  one 
individual  will  be  (presumably)  paid  to  perform  tasks  that  could  be  accomplished 
(in  many  cases  probably  better  and  faster)  by  someone  at  a lesser  salary.  Con- 
versely, the  epitome  of  a specialist  team  approach  would  require  an  educational 
psychologist,  a content  resource  person,  an  instructional  programmer  (both  educa- 
tional logic  and  applications  software),  a key -entry  operator,  and  delivery/evalua- 
tion personnel.  Such  a mix  of  persons  creates  potential  cost  through  A:  a)  idle 
time  due  to  imprecision  in  coordination  or  b)  administrative  expense  to  coordinate 
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the  team,  and  B:  need  for  interaction  between  pairwise  linkage  of  personnel  through 
the  sequence  of  software  generation.  Cost  analysis  can  yield  generalizable  results 
to  indicate  the  efficient  utilization  of  staff. 

(3)  Job  Estimation.  With  widely  varying  degrees  of  precision,  activities  can 
be  analyzed  to  predict  future  costs.  The  similarity  of  future  activities  to  those 
currently  or  formerly  in  operation  plays  a major  role  in  the  precision  of  estimate, 
of  course,  but  a priori  cost  analysis  (in  this  case  an  analysis  of  expected  costs) 
based  on  quantified  parameters  can  and  does  form  the  basis  for  decisionmaking. 

(4)  Personnel  Control.  Whether  the  subject  is  number  of  diodes  quality  con- 
trol tested  per  week,  or  number  of  instructional  programs  generated  per  year,  an 
important  component  of  and  use  for  cost  analysis  is  the  assessment  of  personnel 
performance.  The  latter  case  is  decidedly  more  complex,  but  there  are  at  least 
bandwidth  guidelines  which  can  be  determined  from  past  performance  and  used  for 
comparison  in  the  future. 

(5)  Method  Comparison . The  literature  of  CAI  is  rife  with  appeals  for  or 
attempts  at  comparison  of  CAI  with  Traditional  Instruction  (TI).  Most  efforts  to 
date  are  replete  with  serious  methodological  and  philosophical  problems,  and  be- 
cause of  the  importance  of  this  area  to  ABACUS,  this  topic  will  be  addressed  in 
detail  in  a subsequent  section  of  this  report. 

(6)  Scarce  Resource  Allocation.  In  the  most  general  sense,  resources  and 
costs  are  mutual  inverses,  and  include  space,  time,  personnel,  and  supplies  among 
other  items . When  any  resource  is  in  short  supply  (and  ignoring  for  the  moment 
the  question  of  whether  it  is  available  at  any  "price"),  cost  analytic  procedures  can 
indicate  the  effect  of  allocating  varying  mixes  of  that  resource  among  potential 
recipient  projects.  It  must  be  emphasized  that  this  purpose  of  cost  analysis  is  an 
extreme  sub -set  of  all  cost  analyses,  since  it  is  an  underlying  assumption  tnat  cost 
is  to  be  minimized  (resources  are  to  be  allocated  to  maximize  profit). 

b.  The  Metric  of  Method  Comparison 

One  of  the  most  frequent  uses  of  and  requests  for  cost  analysis  is  generally 
termed  cost/benefit  analysis.  At  the  risk  of  selfish  indulgence,  the  author  is  not 
resisting  the  temptation  to  lapse  into  psychologese  to  speculate  that  administrators 
have  been  searching  for  assistance  in  decisionmaking  and  hope  they  have  found  it 
in  the  (occasionally  spurious)  quantification  of  alleged  benefits.  Assuming  that  all 
costs  (including  space,  faculty  time,  student  time,  equipment,  supplies,  etc.)  have 
been  specified  and  quantified,  what  then  is  the  metric  of  benefit  ? 

(1)  Benefits  Necessarily  Multiple. 

(a)  Edmund  W.  Fitzpatrick,  President  of  the  Educational  Technology 
Center,  proposed  recently  (1973)  that  cost-effectiveness  be  expressed  in  dollar 
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amount  spent  to  achieve  a unit  of  learning  gain.  The  latter  was  defined  as  mean 
posttest  less  mean  pretest.  Such  an  analysis  assumes  equal  intervals  among  units 
and  a defensibly  definable  zero  point  in  the  domain  of  the  unit  metric,  which  topic 
is  potentially  the  subject  of  its  own  monograph,  however.  Fitzpatrick  suggests 
that  it  is  necessary  to  assess  a)  the  cost  per  unit  of  learning  gain,  b)  from  among 
those  alternatives  that  meet  minimum  acceptable  performance  levels,  in  addition 
to  c)  the  acceptability  of  a program  (by  both  faculty  and  students),  and  d)  how  well 
it  is  meeting  its  other  objectives  (he  cites  amount  of  student  time  as  an  example). 
The  point  to  be  drawn  here  is  that  Fitzpatrick  is  avoiding  the  issue  by  proposing 
an  apparently  objective  quantification  scheme,  while  then  augmenting  his  formula 
(dollar  per  unit  of  posttest  less  pretest)  with  border  conditions  ("acceptable"  alter- 
natives, program  "acceptability",  and  "other"  objectives)  that  are  presented  as 
binary.  He  fails  to  account  for  the  variations  among  degrees  of  acceptability,  etc. 
It  is  at  least  obvious  from  these  points  that  the  benefit  side  of  a cost -benefit  equa- 
tion is  complex  and  not  uni -dimensional. 


(b)  Within  the  domain  of  benefits  are  the  immediate  administrative  con- 
cerns of  a)  the  same  x-esults  for  less  cost,  b)  better  results  from  the  same  cost, 
and  c)  the  timing  of  capital  outlay  relative  to  the  accrual  of  results.  To  these  must 
be  added  the  "softer"  benefits  of  a)  student  satisfaction,  b)  greater  assurance  of 
mastery  of  each  student,  and  c)  increased  flexibility  in  the  receipt  of  the  "raw 
materials"  (untrained  students)  and  delivery  of  the  "finished  product"  (graduates). 
Note  that  cost  was  defined  earlier  as  money,  time,  space,  etc.  Hence,  early  de- 
livery of  graduates  (A  t)  is  part  of  "the  same  results  for  less  cost, " and  better- 
trained  graduates  (A  P)  is  included  within  "better  results  for  the  same  cost. " 

(c)  The  softer  benefits  are  more  crudely  measured.  Student  satisfaction, 
for  example,  can  be  intrinsically  measured  by  psychometric  devices  ranging  from 
evaluation  or  appraisal  checklists  to  personality  inventories,  and  its  rather  gross 
extrinsic  measures  include  attrition  from  the  training  program,  poor  on-the-job 
performance,  and  request  for  job  assignment  different  from  that  for  which  the 
graduate  was  trained.  In  a very  real  sense,  these  extrinsic  measures  comprise  the 
"bottom  line"  of  the  training  program,  but  within  locally  defined  reason,  attention 
devoted  to  the  intra-program  satisfaction  measures  may  well  affect  the  extrinsic 
characteristics. 

(2)  Quantitative  Parametric  Comparison 

(a)  Table  1 illustrates  quantitative  aspects  of  cost/benefit  functions. 

The  references  to  traditional  and  technological  instruction  are  included  to  imply 
two  methods  of  which  the  first  is  necessarily  linear  in  cost  (no  start-up  costs, 
fixed  enrollment,  and  static  student -faculty  ratio),  and  of  which  the  second  is 
non-linear. 
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Course  Name:  Basic  Electronics 


Traditional  Instruction: 

A.  Clock  Hours:  210 

B.  Delivery  Expense  (per  student):  $875 

C.  B divided  by  A:  $4.16 

Technological  Instruction: 

D.  Clock  Hours  (average):  165 

E.  Start-up  Costs:  $212,000 

F.  Delivery  Expense  (per  student):  $515 

Assuming  1000  students/year  for  5 years: 

0.  Traditional  Instruction  5 year  cost:  $4,375,000 
(1000  students/year  x 5 years  x $875/student) 

H.  Technological  Instruction  5 year  cost:  $2,787,000 

((1000  students/year  x 5 years  x $515/student)  >($212,000)) 

1.  Technological  Instruction  Cost  per  clock  hour  based  an  5 years, 
1000  students/year : $3.38 

(($2,787,000  divided  by  (1000  students  x 5 years  x 165  clock 
hours/student  (average)  ) 

J.  Cost  percentage  (H  divided  by  G):  63.7% 

K.  Time  percentage  (D  divided  by  A):  78.5% 

L.  Efficiency  Ratio  (H  divided  by  (1000  students/year  x 5 years  x 

210  traditional  hours))  : 2.65 

M.  Traditional  Instruction  (weeks  to  completion):  5.25  (fixed) 

N.  Technological  Instruction  (weeks  to  completion):  3.3  (average) 

O.  Completion  percentage  (N  divided  by  M):  62.8% 
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(b)  Early  in  the  application  of  cost/benefit  techniques  to  instructional 
environments,  the  metric  of  benefit  was  the  student -contact  hour  (SCH).  This 
derived  from  what  Dr.  Warren  Seibert  of  Purdue  University  labeled  the  "sunburn" 
theory:  expose  a student  to  the  golden  (rays  of  sunshine)  words  of  wisdom  for  X 
hours,  and  his  (skin)  learning  behavior  will  change.  The  analogy  of  the  SCH  in 
technological  instruction  must  be  expressed  as  an  average  since  the  process  of 
promoting  a learner  among  stages  in  the  program  heirarchy  at  his  own  pace  neces- 
sitates completion  at  different  times  for  different  students.  The  non-linear  cost 
aspect  of  technological  instruction  is  reflected  in  the  reference  to  start-up  costs. 

(c)  This  distinction  parallels  the  academic  study  of  the  business  world 
in  which  a distinction  is  made  between  labor-intensive  and  capital-intensive  ven- 
tures. The  former  describes  a group  of  87  accountants  working  with  pencil  and 
paper,  and  the  latter  implies  replacement  with  a computer  (large  one-time  capital 
outlay)  and  seven  accountants.  The  first  has  essentially  no  start-up  cost  but 
incurs  a large  cost  per  unit;  the  second  requires  a large  start-up  cost  but  yields 

a smaller  cost  per  unit. 

(d)  Any  comparison  between  the  two  methods  requires  a specification  of 
time  duration.  Based  on  the  data  in  Table  1,  for  example,  if  only  500  students  in 
total  were  involved,  traditional  instruction  would  be  oreferable,  ignoring  other 
cost/benefit  aspects.  500  x $875  ■ $437,500;  (500  x $515)  ♦ $212,000  = $469,500. 
Arbitrarily,  the  analysis  in  Table  1 illustrates  1000  students  and  five  years.  Ele- 
ments A through  F represent  the  source  data  for  comparative  analysis,  and  ele- 
ments G through  O illustrate  the  results  of  manipulations  of  the  source  data  under 
the  assumptions  of  student  rate  per  year  and  duration.  Note  that  for  the  sake  of 
simplicity,  delivery  expense  is  listed  as  a single  dollar  cost  to  aggregate  space, 
faculty  cost,  materials,  etc. 

(e)  The  analogy  of  SCH  for  technological  instruction  can  be  calculated 
only  in  the  presence  of  the  assumptions  (student  rate  and  duration).  Herein  lie 
the  faults  of  the  cost  per  SCH  metric  as  an  index  for  assessing  technological  in- 
struction: a)  five-year  cost  H is  lower,  b)  clock  hours  are  fewer,  and  c)  there 
is  no  stable  rationale  for  comparing  cost/SCH  between  the  two  methodologies 
($4. 16  and  $3.38  in  this  illustration).  More  meaningful  data,  despite  the  fact 
that  they  are  gross  and  macroscopic  and  confound  numeious  variables,  are  the 
cost  and  time  percentages.  But  even  these  are  not  wholly  representative  of  the 
advantage  of  investing  resources  (again,  time,  space,  personnel,  etc.)  in  tech- 
nological instruction. 


(3)  The  Efficiency  Ratio  . There  is  an  implied  commitment  in  the  curriculum 
design  (or,  more  properly,  curriculum  practice)  that  a course  of  210  clock  hours 
does  produce  performance.  Hence,  the  sensitive  metric  of  comparison  of  a tradi- 
tional method  with  a technological  one  needs  to  reflect  the  cost  (of  the  technological 
method)  to  produce  mastery  (as  expressed  in  time  required  by  the  traditional  method). 
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The  elegance  of  this  measure  is  that  it  exemplifies  the  pursuit  of  cost/benefit 
ana'ysis  in  the  ultimate  sense:  What  does  it  cost  to  achieve  a known  benefit? 

Nott  especially  that  thus  far,  this  rather  simple  technique  is  presented  in  the 
context  of  method  comparison  when  both  are  known  and,  presumably  in  operation. 
The  next  section  (c,  below)  will  propose  a projection  of  the  technique  into  environ- 
ments which  are  not  yet  taught  technologically. 

(4)  Another  Quantitative  Measure. 

(a)  The  student  contact  hour  has  received  much  attention  in  public  or 
elective  education  environments  because  the  mandate  of  learning  per  unit  of  time 

is  extremely  less  emphasized  than  in  military -industrial  applications.  In  the  latter 
case,  time  is  truly  of  the  essence  to  some  degree.  Consequently,  it  is  incomplete 
to  base  resource  commitments  solely  upon  a measure  of  clock  hours  to  mastery; 
li»e  of  those  hours  within  the  day  and  week  are  also  of  substantial  im- 

portance. 

(b)  Among  the  reasons  to  justify  technological  instruction  is  the  rather 
glib  statement  that  mediated  (instructor-absent)  learning  environments  allow  the 
student  to  proceed  at  his  own  pace.  The  less -often  identified  administrative  im- 
plication of  that  statement  is  that  when  information  is  presented  by  a machine,  the 
instructor  (who  may  be  eating,  sleeping,  or  enjoying  a well-earned  moment  of 
leisure)  does  not  need  to  be  present.  Thus,  instead  of  testing  the  endurance  of  an 
instructor  beyond  8 hours  per  day  (divided  into  210  (A)  equals  26.25  days,  which, 
divided  by  5 (days  per  week)  equals  5.25  weeks  to  completion  (M)),  it  is  possible 
to  motivate  the  student  to  access  mediated  instructional  materials  50  hours  per 
week  (divided  into  165  (D)  equals  3.3  weeks  to  completion  (N))  at  times  he  elects. 
Thus,  another  quantitative  measure  of  the  benefit  of  technological  instruction  is 
weeks  to  completion  and,  comparatively,  the  completion  percentage  (O).  The 
supply  of  raw  materials  (students)  in  a timely  manner  and  the  shipment  of  the 
finished  product  (graduates)  are  subjects  worthy  of  receiving  attention  elsewhere 
(but  are  not  addressed  in  this  report). 

c.  Estimating  Future  Applications 

The  techniques  discussed  in  the  prior  section  address  comparisons  between 
existing,  presumably  parallel  methodologies.  How  can  the  cost/benefit  parameters 
of  technological  instruction  be  estimated  in  the  absence  of  a full  commitment  to 
establish  it? 

(1)  The  Traditional  Method  Exists.  Within  rather  loosely-defined  guidelines, 
it  is  possible  to  analyze  the  content  and  objectives  of  instruction,  and  to  compare 
the  abstract  features  of  an  existing  traditional  method  with  those  similar  features 
which  characterize  a technological  effort.  By  implication  of  similarity,  generali- 
zations can  be  made  based  on  the  extent  and  nature  of  the  effort  devoted  to  the 
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former  technological  development,  and  these  approximations  will  provide  an  initial 
estimate  of  the  probable  costs  and  benefits  of  translating  the  traditional  instruction 
into  a technological  format.  Table  2 illustrates  one  possible  taxonomy  of  abstract 
instructional  characteristics.  Because  the  nature  of  such  an  analysis  is  highly 
content -specific,  no  further  elaboration  will  be  made.  Fairly  precise  estimation 
can  be  performed  independent  of  the  potential  benefit  of  this  comparison  of  abstract 
characteristics,  however;  a proposed  technique  for  this  estimation  is  the  subject  of 
the  next  section  I,  c,  (2). 

(2)  The  Traditional  Method  Does  Not  Exist . Aye,  here's  the  rub.  When 
approaching  an  instructional  task  de  novo,  there  exists  the  choice  of  implementation 
in  a traditional  or  a technological  format . Since  the  early  development  stages 
(identifying  entry-level  behavior,  determining  learner  aptitude,  content  sequence, 
structure,  and  task  analysis)  should  be  identical  whether  either  teaching  method  is 
known  (or  specified)  a priori,  the  basic  lesson  plan  will  be  established.  If  the 
leverage  (student  rate  and  duration)  is  great,  simulations  of  each  method  are  well 
worth  the  time  and  expense.  What  is  being  proposed  here  is  the  development  of  a 
simulation  of  each  method,  using  a small  number  of  students. 

(a)  Simulate  the  Traditional  Method. 

1_  Microteaching  was  proposed  by  Dr . Kevin  Ryan  of  the  University  of 
Chicago  (the  initial  literature  was  published  while  he  was  at  Stanford  University), 
and  has  received  wide  acclaim  primarily  as  a technique  for  training  primary  and 
secondary  level  teachers.  An  unexpected  spin-off,  however,  was  the  realization 
thrf  uhe  learning  performance  of  the  students  was  a confounded  (in  the  sense  of 
unseparated  variables)  effect  of  both  the  instructor  and  the  lesson  plan.  Hence, 
with  a proven  lesson  plan,  a novice  instructor  can  be  evaluated,  and  with  an 
experienced  instructor,  an  experimental  lesson  plan  can  be  evaluated. 

2 The  basic  technique  of  microteaching  involves  a physical  setting 
essentially  identical  to  the  real-life  classroom  environment,  and  with  a class 
size  of  six  to  eight  students  that  represent  a stratified  sample  of  the  student  popu- 
lation. In  the  teacher  training  application,  "classes"  are  split-screen  videotaped 
with  one  camera  on  the  class  and  another  on  the  student  teacher.  The  tapes  docu- 
ment the  class  sessions  for  critique  by  an  education  faculty  member  in  the  pre- 
sence of  the  student.  For  the  purpose  of  the  simulation  proposed  here,  an  upper 
bound  of  student  time  to  reach  mastery  can  be  established  with  trial  handout  ma- 
terials and  orientation  of  the  instructor  to  "pretend"  that  he  is  teaching  to  a larger 
class  size.  This  exercise  will  yield  not  only  an  upper  bound  of  time  (upper  bound 
because  of  the  presumption  that  materials,  sequence,  etc. , will  be  refined  for 
successive  presentations  of  the  course),  but  valuable  data  for  course  revision 
whether  finally  delivered  (to  students)  traditionally  or  technologically.  Analysis 
of  this  simulation  will  yield  estimates  similar  to  A through  C in  Table  1 for  use  in 
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1.  This  author  has  been  asked  to  exemplify  computer  assisted  in- 
struction in  the  absence  of  a computer.  In  many  a hotel  corridor  and  airport  limou- 
sine, the  game  has  been  played  as  follows: 

Skeptic:  Don't  put  me  on! 

Fitzgerald:  O.  K. , I'll  play  the  part  of  the  computer  and  you  answer 
any  way  you  want;  do  you  know  what  the  successor  of  a 
number  is  ? 

Skeptic:  Sure,  it's  the  next  number  higher. 

Fitzgerald:  Please  answer  "yes"  or  "no". 

Skeptic : Yes 

Fitzgerald:  Good,  but  I am  going  to  ask  you  to  prove  it;  what  is 
the  successor  of  3 ? 

Skeptic : 4 

Fitzgerald:  Good.  Now  tell  me,  what  is  8 times  its  own  successor? 

Skeptic:  Huh?  You  mean  8 times  9? 

Fitzgerald:  I'm  sorry,  I did  not  understand  your  answer.  I am  programmed 
to  expect  a numerical  response  to  this  question;  please  try  again. 

etc. 

2 Despite  the  raised  eyebrows  one  receives  from  strangers  who  walk 
into  the  middle  of  this  simulation,  it  does  make  its  point.  What  is  being  proposed 
is  an  unpolished  sequence  of  technological  modules  that  present  an  upper  bound 
(in  time)  estimate  of  the  learning  of  a representative  sample  of  students  (in  the  best 
design  pairwise  matched  with  the  characteristics  of  the  student  sample  proposed  in 
I.c,  (b),  1^  above.)  Quick  videotapes  can  now  be  made  with  a Sony  Porta-pak;  trained 
personnel  can  simulate  an  instructional  computer  program  (note:  tape  record  the 
conversation  for  unanticipated  response  data  to  be  used  in  the  CAI  programming 
effort);  and  first -approximations  of  slide/tape  programs  can  be  quickly  prepared  in 
storyboard  format  with  a Polaroid  camera. 
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Table  2.  DETERMINANTS  OF  THE  CHOICE  OF  LEARNING  ENVIRONMENTS 
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1.  Nature  of  the  content : verbal,  still  visual,  moving  visual,  auditory,  positional, 

etc. 

2.  Nature  of  the  response:  written,  spoken,  behavioral,  etc. 

3.  Time  available:  a)  for  the  acquisition  of  the  behavioral;  i.  e. , perishability  of 

the  stimuli,  b)  for  the  demonstration  of  the  response. 

4.  Problems  to  solutions: 

A.  content:  verbal 

response:  written  or  spoken — direct  repetition  from  memory 
duration  of  stimulus:  variable 
time  allowed  for  response:  variable 

SOLUTION : Printed  text 

B.  content:  verbal  and  positional  (partial  cues) 

response:  written,  spoken,  positional — demonstration  by  analogy 
duration  of  stimulus:  variable  within  constraints 
time  allowed  for  response:  variable  within  constraints 

SOLUTION:  Lecture 

C.  content:  verbal  ani  still  visual 

response:  written,  spoken,  positional — demonstration  by  repetition  or 
analogy 

duration  of  stimulus:  variable 

time  allowed  for  response:  variable  within  constraints 
SOLUTION:  Slide/tape  (text,  still  pictures) 

D.  content:  verbal  and  moving  visual 

response:  written,  spoken,  positional — demonstration  by  repetition  or 
analogy 

duration  of  stimulus:  fixed  but  replicable 

time  allowed  for  response:  variable  within  constraints 

SOLUTION:  Videotape  (movies) 

E.  content:  verbal,  auditory,  positional 

response:  verbal,  positional — demonstration  by  repetition  or  analogy 
duration  of  stimulus:  fixed  but  not  critical 


Table  2.  DETERMINANTS  OF  THE  CHOICE  OF 
LEARNING  ENVIRONMENTS  (Cont) 

content:  auditory,  positional 

response:  verbal,  positional — demonstration  by  repetition  or  analogy 
duration  of  stimulus:  fixed  and  critical 
time  allowed  for  response:  fixed  and  critical 

SOLUTION:  "Live* interaction — ultimate  experience 
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(3)  Penny  Foolish  and  Pound  Wise. 

A program  of  method  simulation  may  draw  skeptical  and  critical  remarks 
from  colleagues  and  financial  officers,  but  if  the  leverage  of  the  effort  is  substantial, 
the  potential  cost  saving  may  grossly  outweigh  the  simulation  expense.  The  con- 
duct of  such  a parallel  program  may  receive  well-founded  criticism  that  each  sim- 
ulation is  at  best  only  a crude  approximation  of  the  potential  end  product,  but  the 
basic  assumption  here  is  that  the  methods  are  proportionately  upper  bounds,  and 
that  it  is  the  time  and  expense  ratio  that  is  of  interest. 


Section  n.  Miscellaneous  Comments 

a.  Instructional  Programmer  Training . 

During  the  December  1974  site  visit,  the  point  was  made  that  the  structure  of 
the  instructional  applications  software  was  sufficient  to  "orient"  untrained  CAI 
program  authors  to  the  methodology  of  instructional  program  logic . Speaking 
ex  cathedra,  this  author  submits  that  many  paradigms  of  instructional  logic  have 
been  developed  through  experience,  and  an  illustration  of  prototype  examples  may 
inspire  novice  authors  to  invent  variations,  and  at  least  may  save  author  program 
generation  time  by  avoiding  a re-invention  of  the  proverbial  wheel. 

b.  The  Uses  of  Affective  Data. 

The  relationship  between  personality  attributes  and  type  of  learning  environ- 
ment is  under  investigation  in  a wide  variety  of  diverse  environments,  but  explor- 
atory findings  indicate  that  some  relationships  may  exist.  Since  the  ABACUS  Pro- 
ject may  be  less  subject  to  invasion  of  privacy  (OMB)  restrictions  than  other  en- 
vironments, it  may  be  highly  profitable  to  experimentally  collect  personality  data. 

(1)  Student  Selection/Reiection.  Students  selected  at  random  to  participate 
in  instructional  experiments  have  reported  affective  reactions  to  different  environ- 
ments. Some  students  have  reported  concern  over  personal  initiative,  study  habit 
compulsiveness,  and  self -scheduling  laxity.  Some  of  these  variables  may  predict 
success  and  failure  in  self-paced  environments  and  profiles  of  affective  traits 
could  predict  failure  (resulting  in  cost  waste). 

(2)  Program  Modification.  This  author  defines  the  goal  of  instructional 
systems  technology  as  the  optimization  of: 

(a)  the  greatest  learning, 

(b)  in  the  shortest  time. 
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(c)  with  the  greatest  long-term  retention,  and 

(d)  with  negative  affective  reactions  within  threshold  boundaries 

It  would  be  conceivable,  although  highly  undesirable,  to  subject  learners  to  an  in- 
structional situation  in  which  the  reward  for  performance  was  modest,  but  the  pun- 
ishment for  failure  was  cruel.  The  first  three  parameters  of  the  optimization 
function  above  could  be  fulfilled,  but  at  the  expense  of  anxiety,  resentment,  and 
hostility.  The  moral  issue  may  be  obvious  and  a question  of  degree,  but  as  a prac- 
tical matter,  unresolved  negative  affect  is  frequently  displaced  symbolically  both 
consciously  and  unconsciously.  Since  the  expression  of  displaced  agression  is  often 
insidious  and  well  disguised,  even  from  the  subject  himself,  it  may  well  behoove 
the  Project  to  assess  the  affective  consequences  of  technological  training  during  a 
course . 
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NOTE:  For  a well -presented  illustration  of  a post  hoc  accounting  method, 

do  not  be  distracted  by  the  content  application,  and  examine  a copy  of: 

Cost  Analysis  and  Rate  Setting 
Manual  for  Animal  Resource  Facilities 
available  from: 

Animal  Resources  Branch 
Division  of  Research  Resources 
National  Institutes  of  Health 
Bethesda,  Maryland  20014 

Telephone:  (301)  496-5507 
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Richard  C.  Horne,  in  (USA  Ret.  ) 


Richard  C.  Horne,  in.  Brigadier  General  USA  (Ret.)  was  born  in  Savannah, 

GA,  on  21  July  1920.  He  attended  public  schools  in  S.  C.,  D.  C.  and  VA.  He 
graduated  from  Virginia  Military  Institute  in  May  1942  with  a B.  S.  in  Electrical 
Engineering.  He  entered  the  Army  in  May  1942.  He  served  in  the  Pacific  Theater 
from  Dec.  1943  until  Dec.  1945,  culminating  as  company  commander  of  the  38th 
In£  Div.  Sig.  Co.  He  served  in  various  staff  and  command  duties  until  1949.  Then 
he  was  assigned  for  four  years  to  Sandia  Base,  N.  M.  on  special  weapons 

Following  a tour  in  Korea  with  the  Signal  Section  of  Eighth  Army,  he  served 
for  almost  two  years  in  Joint  Military  Headquarters  in  Japan.  BG  Horne  returned 
from  Japan  and  attended  the  Command  and  General  Staff  College  at  Fort  Leaven- 
worth for  a year.  Next,  he  served  for  three  years  at  the  North  American  Air 
Defense  Command  in  Colorado  Springs.  Next,  he  had  a one  year  tour  at  the  Naval 
War  College  in  Newport,  Rhode  Island. 

In  the  summer  of  1961,  BG  Horne  commenced  a two  and  one-half  year  tour  of 
Izmir,  Turkey,  as  the  officer-in -charge  of  a 700-man  multinational  (7  services, 

3 nations)  communications  facility  utilizing  large  fixed-station  equipment  as  well 
as  hundreds  of  mobile  equipments. 

From  Turkey,  he  was  assigned  as  Signal  Officer,  III  Corps,  participating  in 
the  huge  Desert  Strike  maneuver  managing  tactical  communications  for  an  inde- 
pendent armored  corps  in  a maneuver  covering  thousands  of  square  miles  of 
desert  and  mountain. 

The  ni  Corps  provided  the  first  Corps  type  headquarters  for  Vietnam  in  1965. 

He  served  as  Signal  Officer  in  the  central  region  of  South  Vietnam  managing  tactical 
communications  over  an  area  as  large  as  South  Carolina  under  stress  conditions 
during  the  early  battles  and  build-up. 

Following  Vietnam,  he  served  two  years  as  the  Director  of  the  Communications 
Electronics  Department  of  the  Infantry  School  engaged  in  teaching  of  tactical  com- 
munications to  thousands  of  Infantry  Officers  and  soldiers. 

Commencing  in  1968,  he  served  as  Commandant  of  the  Signal  School  at  Fort 
Monmouth.  This  school  reached  strengths  (military  and  civilian)  of  over  13,000. 
After  more  than  thirty  years  of  service,  he  retired  as  Brigadier  Cjneral  in 
December  1972. 

From  1 April  1973  until  1 April  1974,  BG  Horne  worked  as  Sr.  Research  En- 
gineer at  Stanford  Research  Institute.  He  was  involved  with  studies  and  proposals 
for  research  in  tactical  and  strategic  military  communications,  national  and  local 
disaster  communications,  public  safety  communications,  and  the  use  of  computers 
in  technical  training. 
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CTS  in  the  Production  and  Utilization  of  Technical  Specialized  Skills 

by 

Richard  C.  Horne  in 
Brigadier  General  USA  (Ret . ) 

General 

a.  The  United  States  Army  has  a growing  need  for  increasingly  numerous 
categories  of  technical  specialized  skills.  This,  paradoxically,  has  small  re- 
lationship to  the  size  of  the  Army,  but  is  rather  a reflection  of  the  complexity 
of  modern  life  and  its  technology.  If  we  are  to  have  modern  weapons  systems, 
we  must  have  the  technicians  to  operate  and  maintain  them.  Their  categories 
(Military  Occupational  Specialities,  or  MOS)  increase  directly  as  the  complexity 
of  the  machines  increases.  This  problem  is  not  completely,  nor  even  signifi- 
cantly, of  the  Army's  own  making,  so  "simplifying  the  Army"  or  "let's  get 
back  to  basics"  is  not  the  answer. 

b.  The  answer  is  clear,  but  not  simple,  not  easy,  and  not  cheap.  It  is  — 
quickly  recognized  valid  requirements  for  skilled  technicians  — produce  them  to 
adequate  standards  as  expeditiously  as  possible,  and  utilize  them  effectively. 

c.  The  Army  has  recognized  this  requirement  and  is  examining  methods  to 
fulfill  it.  CTS  is  one  method,  and  seemingly  one  with  the  potential  to  fulfill  it 
well,  if  exploited  to  the  full.  It  is  necessary,  therefore,  to  examine  CTS  as  a 
part  of  a system  which  produces  and  utilizes  technical  specialized  skills — and 
not  look  on  it  as  merely  a novel  and  exciting  way  to  use  computers  to  teach. 

The  Production  Phase 

a.  Production  of  technical  specialists  consists  of  recruiting,  selecting,  and 
training  of  individuals . These  steps  are  further  complicated  by  a limited  supply 
of  raw  material,  politics,  confusion,  lack  of  money,  etc.  These  are  formidable 
hindrances  but  apply  to  virtually  any  large  complex  endeavor  in  a modern  country. 
Obviously,  then,  the  production  system  must  operate  in  spite  of  these  hindrances, 
with  minimum  sensitivity  to  them. 

b.  Recruiting. 

(1)  Although  the  Army's  technical  specialists  are  certainly  in  the  minority, 
the  "glamour"  of  these  specialities  is  widely  used  to  attract  enlistment.  We  are 
a nation  of  people  living  in  close  affinity  with  our  tec  hi  .tlogy,  and  our  youth,  re- 
gardless of  technical  aptitude,  are  inclined  to  respect  technical  things  and  skills 
as  producers  of  status,  a quality  few  prospective  recruits  have. 

Preceding  page  Hank 
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(2)  The  recruiter,  utilizing  the  salesman's  tools  of  brochures,  visual  aids, 
and  just  plain  salesmanship,  endeavors  to  interest  the  prospect  in  particular  MOS's 
the  Army  needs.  If  he  is  successful,  the  prospect  is  given  an  aptitude  test  battery, 
physical  examination,  enlists,  and  signs  a contract  for  training  in  the  skill  he 
wants  and  the  Army  needs . 

(3)  The  test  battery  is  a gross  sorter  of  aptitudes  and  basic  intelligence.  It 
is  not  a precise  predicter  of  success  in  a particular  technical  skill.  Likewise,  the 
physical  examination.  For  example,  a color-blind  man  cannot  enlist  for  most 
electronic  maintenance  skills  due  to  his  inability  to  perceive  color  coding  of  cer- 
tain basic  electronic  components . The  important  thing  to  remember  about  the 
recruitment  phase  in  this  instance  is  that  by  gross  sorting  the  Army  bets  that  this 
recruit  can  be  made  into  a certain  type  of  technician.  If  it  fails,  or  he  fails,  the 
Army  must  train  him  in  some  other  MOS  of  the  Army's  choosing.  If  the  Army 
makes  an  error  and  overlooks  certain  disqualifying  scores  on  the  test  battery  or  a 
physical  defect,  and  the  recruit  fails  in  bis  skill  training  for  reasons  associated 
with  this  overlook,  then  the  recruit  may,  at  his  option,  seek  discharge  or  other 
training.  There  Is  a need,  then,  for  as  accurate  aptitude  determination  as  possible 
before  the  Army  contracts  to  give  skill  training,  often  at  a cost  of  many  thousands 
of  dollars,  to  a recruit,  ft  seems  reasonable  that  CTS,  in  its  CMI  portion,  could 
produce  feedback  valuable  to  the  test  battery,  allowing  a more  refined  use  of  it. 
Perhaps  better  yet,  CTS  feedback  data  may  facilitate  the  design  of  an  aptitude 

test  designed  for  the  MOS  desired  by  the  recruit,  to  be  given  later  in  the  recruit- 
ing process. 

c.  Basic  and  Skill  Prerequisite  Training. 

(1)  Basic  training  teaches  the  basic  soldiering  skills  to  all  enlistees,  ft 
has  no  direct  connection  with  CTS  except  to  act  as  a further  screening  process . 

At  one  site  of  basic  training,  certain  graduated  basic  trainees  get  training  in 
basic  electronics  (COBET)  before  being  sent  to  service  schools  for  skill  training 
in  MOS's  requiring  electronics  as  a prerequisite.  The  amount  of  electronics 
training  varies  with  the  knowledge  of  electronics  required  in  the  forthcoming 
technical  skill  to  be  taught  at  a service  school. 

(2)  This  concept  is  a money-saver.  No  travel  is  involved.  Experience  has 
shown  that,  in  the  electronic  skills,  most  of  the  attrition  takes  place  in  the  early 
phases  of  the  course,  which  is  the  basic  electronics  portion,  ft  is  expensive  to 
transport  a soldier  a thousand  miles  to  a service  school,  only  to  have  him  fail  in 
the  first  few  weeks  and  have  to  be  transported  to  another  service  school  or  training 
center  for  alternate  training.  CTS  must  act  as  a feedback  to  this  skill  prerequisite 
training,  refining  the  requirements  for  entry  into  the  CTS  course,  and  hopefully, 
lowering  attrition. 
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(3)  There  seems  to  be  no  reason  why  other  skill  prerequisites  common 
courses  cannot  be  developed  to  take  place  at  the  basic  training  site.  A course  in 
basic  mechanical  skills,  for  example,  for  those  entering  mechanical  MOS  training. 

K seems  entirely  clear,  however,  that  the  skill  courses  of  CTS  must  feed  back  to 
the  skill  prerequisite  courses,  and  close  coordination  maintained  among  those  who 
write  these  courses . 

d.  Technical  Skill  Training  at  Service  Schools . 

(1)  The  Service  schools  receive  their  trainees  from  the  basic  training  site, 
gross-sorted  by  the  initial  aptitude  test  battery,  the  physical  examination,  and  basic 
training.  For  those  who  have  had  skill  prerequisite  training,  a screening  has  taken 
place.  This  screening  gives  the  service  school  (CTS  in  this  case)  a probability  of 
completing  the  training  consistent  with  the  validity  of  its  process  of  date  exchange 
and  evaluation  with  previous  test  and  training  points.  The  point  is,  CTS  can  par- 
ticipate in  selecting  who  enters  its  courses . 

(2)  The  conduct  of  CTS  training  will  not  be  dealt  with  here  except  to  say  that 
its  continuous  evaluation  and  prediction  functions  are  vital  in  the  individual's  on- 
going training  and  the  Army's  management  of  its  resources  and  requirements  in  the 
skill  produced  by  the  CTS  course. 

e.  On-the-Job  Training  (OJT)  and  Initial  MOS  Validation. 

(1)  CTS  has  at  this  stage  produced  an  apprentice  technician  in  a time  con- 
sistent with  his  individual  ability,  and  has  to  some  degree  evaluated  his  potential. 
The  Army,  based  on  the  earliest  valid  prediction  of  his  graduation  date,  has 
arranged  to  send  him  to  a unit  where  utilization  of  his  skill  commences.  Although 
he  is  in  a unit,  and  may  be  participating  in  unit  training,  an  important  part  of  his 
individual  training  remains.  This  is  OJT,  designed  to  raise  him  to  the  journeyman 
level,  working  in  the  environment  particular  to  unit  utilization  of  his  skill. 

(2)  The  service  school  (CTS)  must  provide  the  technical  design  of  his  OJT. 
CTS  offers  the  potential  of  an  OJT  training  package  designed  to  the  actual  require- 
ments of  the  individual,  rather  than  one  applicable  to  the  "average"  graduate  of 
the  skill  course.  CTS  should  be  able  to  vary  the  suggested  OJT  package  for  a 
particular  individual  based  on  the  evaluation  of  his  service  school  performance, 
together  with  feedback  data  from  the  unit  on  former  graduates  of  the  school . The 
unit  administers  OJT  to  the  individual  and  on  completion,  subject  to  the  judgment  of 
the  unit  commander,  makes  the  initial  validation  of  his  MOS  as  a journeyman.  Data 
on  the  individual's  performance  during  OJT  must  be  fed  back  to  CTS  in  order  to 
improve  its  teaching,  evaluation,  and  prediction  functions . 
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(1)  After  the  technic  .an  has  achieved  journeyman  status,  CTS  has  little  to  do 
with  his  utilization  except  periodic  MOS  validation.  The  CTS  course  must  be  the 
basis  of  MOS  validation,  as  it  is  the  only  measurable  expression  of  the  detailed 
technical  skills  involved.  It  is,  or  should  be,  the  initial  point  where  new  equip- 
ment or  techniques  are  introduced  into  the  MOS.  It  should  be  the  source  of  up- 
dating information  or  training  material  sent  to  the  field.  It  follows,  then,  that 

the  CTS  should  prepare  M06  validation  for  journeymen  in  the  field,  whether  it  be 
a purely  "go-no-go"  qualification,  or  an  evaluated  validation,  which  seem,  more 
desirable. 

(2)  It  would  be  even  more  desirable  that  periodic  MOS  validation  be  achieved 
"on  line"  at  the  service  school,  either  by  the  physical  presence  of  the  technician, 
or  by  other  means  of  access  to  the  CTS  computers  and  equipment. 

g.  Up-grading  Skills. 


(1)  The  technician,  if  he  remains  in  the  Army,  must  go  "up  or  out.  " He 
must  achieve  higher  levels  of  skill  in  his  journeyman  MOS,  or  he  must  pass  to 
different  MOS's  using  his  original  skill  in  addition  to  others  learned  by  experience 
or  additional  formal  training.  CTS,  as  discussed  so  far,  deals  with  individual 
MOS-producing  courses.  It  is  logical,  however,  that  higher  or  broader  skill  MOS's 
can,  in  technical  content,  be  derived  by  combining  appropriate  sections  of  several 
lower-skill  CTS  courses.  Essentially,  this  creates  another  CTS  course  for 
screened  and  specially  selected  students.  Feedback  from  the  field,  particularly 
that  produced  from  continuing  MOS  validation  would  be  essential  in  the  construction 
of  the  higher -level  course  and  the  selection  of  those  to  be  trained.  For  those  who 
see  inherent  evil  in  "promotion  by  computer,  " those  safeguards  guaranteed  by  the 
subjective  judgment  of  commanders  are  not  threatened,  as  they  are  largely  inde- 
pendent of  the  training  system. 


(2)  At  the  higher  levels  of  training,  a great  many  follow-on  actions  are 
simplified.  OJT,  as  such,  need  not  be  required.  Only  MOS  validation  from  the 
technical  standpoint,  and  feedback  into  the  training  system  for  evaluation  and  pre- 
diction are  necessary.  The  technician  is  advancing  to  the  supervisory  level  where 
broader  and  more  subjective  judgments  determine  his  future. 

h.  Interrelationship  of  Training  and  Personnel  Management  ■ 

(1)  Previous  paragraph  have  alluded  to  information  produced  by  the  training 
system  and  utilized  by  personnel  management  at  the  highest  Army  level. 
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(2)  Simply  stated,  the  Army  personnel  managers  must  know: 

. What  technical  skills  are  needed 

. How  many  of  each 
. Where  to  put  them 
. When  are  they  available 

(3)  CTS  is  not  a candidate  source  for  complete  ans  vers  to  those  basic 
questions.  CTS,  however,  does  appear  to  be  the  only  system  in  sight  that  in- 
cludes functions  capable  of  producing  information  to  up-grade  the  Army's  esti- 
mation of  the  answers . 

(4)  Factors  beyond  the  Army's  ability  to  influence  do  have  large  affect  on 
its  ability  to  answer  the  questions.  However,  accurate  and  timely  prediction  of 
the  training  system's  output  can  provide  a dramatic  improvement  to  personnel 
management  and  save  many  millions  of  dollars . 

(5)  Basically,  the  Army  should  know,  with  the  greatest  possible  accuracy: 
How  many  recruits,  of  what  ability,  must  it  take  to  produce  x number  of  appren- 
tice technicians  at  y time  ? It  must  know  this  for  each  skill  it  needs . & very  gross 
terms,  based  on  empirical  data,  the  Army  knows  this  now.  However,  the  answer 
is  unsatisfactory.  Too  many  recruits  are  required,  and  stay  too  long  in  the  train- 
ing pipeline,  and  too  few  come  out.  Changing  these  factors  beneficially  could  mean 
billions  of  dollars  in  eventual  savings . 

(6)  It  is  believed  that  CTS  can  provide  the  following  benefits , if  properly 
exploited  as  a system  touching  on  every  aspect  of  the  Army's  training  mission, 
and  closely  interrelated  with  personnel  management: 


l 
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. Refine  aptitude  standards  for  better  prediction  of  success 

. Determine  the  desirability  of  special  aptitude  tests  for 
certain  high-cost  skills 

. Improve  the  course  content  of  skill  prerequisite  courses 

. Predict  and  report  at  entry  into  skill  course  (CTS)  probability 
of  completion 

. Predict  and  report  within  skill  course,  date  of  completion 

. Recommend  on  an  individual  basis  type  and  duration  of  OJT 
to  unit  commander 
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. Provide  MOS  validation  at  any  time 

. Predict  skill  up-grading  potential  of  individual 

. Construct  combined  high  skill -level  courses  and  resulting  MOS 
validation 

. Continue  to  evaluate  and  utilize  data  collected  to  improve  and  modify 
related  training  processes 


i.  CTS  Applicability  to  General  Knowledge  Courses  . 

CTS  can  be  used  for  general  knowledge  type  courses  (mathematics,  para- 
professional  legal  or  medical  subjects,  etc).  The  benefits,  actual  and  potential, 
appear  the  same,  particularly  the  evaluation  and  prediction  functions.  The  advan- 
tages of  self-pacing  and  individually  prescribed  remedials  are  obvious . The  main 
question  seems  to  be  one  of  cost-effectiveness.  If  the  general  knowledge  courses 
are  vital  to  the  Army's  needs  and  those  needs  cannot  be  fulfilled  by  conventional 
methods  in  a timely  and  effective  manner,  then  CTS  would  appear  a strong  candi- 
date to  perform  the  mission. 

j.  Applicability  to  Reserve  and  National  Guard  Training . 

(1)  CTS  is  ceitainly  applicable  to  Reserve  and  National  Guard  training, 
provided  those  components  use  the  same  equipment  ag the  active  Army. 

i 


(2)  CTS  offers  the  following  advantages  to  Reserve  and  National  Guard 
training: 

. Standardized  training  for  all  Army  components  using  the 
same  equipment 

. The  ability  to  insert  new  trainees  into  the  program  at  any  time 

. The  ability  to  pace  the  training  to  any  schedule  of  trainee 
availability 

. The  potential  for  off-site  training  utilizing  leased  circuits  for 
access  to  computer  portions  of  the  program.  Reserve  and 
National  Guard  training  usually  takes  place  at  night  and  on 
weekends . This  facilitiates  the  use  of  CTS  computers  and 
raises  their  cost  effectiveness. 

. The  potential  for  increased  effectiveness  in  the  management  of 
Reserve  and  National  Guard  training. 
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(3)  One  factor  should  not  be  overlooked.  The  National  Guard,  and  to  some 
lesser  extent,  the  Reserve,  are  potent  political  forces.  They  have  lambasted  the 
active  Army  for  years,  claiming  second-class  status  in  training  and  equipment 
and  with  reason.  Anything  to  improve  this  genuine  imbalance  would  be  not  only 
a political  help  to  the  active  \rmy  but  do  much  to  improve  the  nation's  military 
position. 

k.  Summary . 

(1)  CTS  must  be  looked  at  as  a part  of  the  Army  training  system  impacting 
on  recruiting,  training,  irtilization  and  management  of  technical  specialists. 

(2)  Its  benefits  can  be  stifled  if  it  is  used  only  as  a training  method,  even 
though  it  is  a most  effective  training  method.  Its  cost-effectiveness  improves 
dramatically  when  its  CM3  aspects  (data  gathering,  evaluation,  prediction,  self- 
improvement,  etc.)  are  exploited.  Its  contributions  to  more  precise  management 
of  scarce  technical  skill  resources  can  be  dramatic. 

(3)  All  these  benefits,  however,  can  be  derived  only  if  it  is  developed  and 
utilized  as  a dynamic  system  with  all  its  functions  interrelated,  each  affecting 
and  being  affected  by  the  other. 

(4)  The  analogy  of  the  wheel  in  motion  seems  particularly  appropriate. 

It  is  a dynamic  structure,  its  progress  determined  by  movement  of  its  central 
number,  with  all  other  structural  members  continuously  reacting  with  not  only 
the  central  member  but  with  each  other. 
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Reactions  and  Suggestions  on  U.  S.  Army 
Computerized  Training  System 

Harold  E.  Mitzel,  Pti.  D. 

January  10,  1975 

The  purpose  of  this  report  is  to  respond  to  the  invitation  to  comment  on  CTS 
which  was  extended  to  the  Panel  of  Consultants . In  the  first  section  I will  indicate 
some  general  impressions,  mostly  positive,  based  on  the  site  visit  at  Fort  Gordon, 
Georgia,  December  7-8,  1974.  In  the  second  section  I have  tried  to  respond  to 
selected  questions  posed  to  the  Systems  and  Training  Groups . In  the  final  third 
section,  I have  made  a recommendation  for  the  future  development  of  CTS. 

General  Impressions 

a.  This  project  seems  to  be  essentially  on  its  scheduled  developmental  track 

in  spite  of  delays  by  the  hardware  and  software  supplier.  I was  favorably  impressed 
with  the  Sylvania  cathode -ray -tube  terminal  in  relation  to  its  cost.  It  does  appear  to 
have  sufficient  "bells  and  whistles"  to  interface  the  management  of  the  three  tech- 
nical repair  courses  involved  in  the  current  effort.  I would  hope  that  the  next  gen- 
eration of  equipment  would  include  an  audio  capability  under  program  control.  The 
cluster-plan  for  organizing  the  hardware  configuration  with  its  sharing  of  course- 
ware storage  is  esse-tially  sound  and  should  yield  minimal  queuing  and  rapid  re- 
sponse for  students. 

b.  I was  particularly  impressed  with  the  knowledgeability  and  dedication  of  the 
CTS  staff,  both  civilian  and  uniformed.  The  morale  of  the  staff  seemed  good  at  all 
levels,  but  was  striking  among  the  NCO's.  They  had  obviously  worked  hard  to  bring 
the  existing  system  up  to  its  maximum  potential  for  the  consulting  panel's  review. 
They  were  well -acquainted  with  the  features  of  the  hardware  and  with  the  overall 
plan  for  CTS . 

c . The  evaluation  plan  for  CTS  needs  to  be  mentioned  in  that  the  Consultant 
Panel  spent  a great  deal  of  time  discussing  it  with  Mr . Longo . My  general  im- 
pression was  that  the  Panel  was  highly  critical  of  the  evaluation  plan.  However, 
reviewing  groups  (particularly  when  made  up  of  experts)  frequently  misunderstand 
planning  efforts  because  they  are  unaware  of  crucial  assumptions  and  key  situational 
factors  over  which  the  evaluation  planners  have  little  or  no  control . After  studying 
the  Preliminary  Evaluation  Plan  for  CTS  and  taking  into  account  the  shift  in  location 
from  Ft.  Monmouth  to  Ft.  Gordon  with  its  attendant  change  in  emphasis  from  CAI 
to  CMI,  I think  the  plan  is  basically  a good  one. 

Preceding  page  blank 
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d.  ft  does  seem  to  me  that  perhaps  too  great  an  expectation  has  been  placed  on 
the  so-called  summative  evaluation,  ft  is  a mistake  for  the  Army  to  make  a global 
generalized  decision  about  the  potential  of  computers  in  training  chiefly  on  the  basis 
of  the  present  effort.  In  the  first  place,  this  effort  deals  only  with  three  specialized 
equipment  repair  courses . If  one  wished  to  generalize  to  the  wide  variety  of  training 
offered  in  the  Army,  one  would  need  to  carefully  sample  courses  from  some  identi- 
fiable universe  of  courses  for  any  evaluative  effort  to  be  generalizable.  ft  seems 
patently  obvious  that  computers  applied  to  instruction  are  going  to  work  better  in 
some  subject  matter  areas  than  in  others.  Secondly,  a computerized  course  is 
complex  and  inevitably  composed  of  many  different  elements . These  elements  are 
inextricably  tied  together  as  weighted  causative  factors  in  the  measured  output . 

Some  of  the  factors  which  exert  a potentially  powerful  influence  on  the  overall 
quality  of  the  output  from  CTS  include: 

(1)  the  pedagogical  strategy  employed  by  the  instructional  programmers 
(authors). 

(2)  the  responsiveness  and  interactive  speed  of  the  terminals. 

(3)  the  quality  of  the  answer  processing  provided  by  the  operating  system. 

(4)  the  mix  of  "on-line"  and  "off-line"  instruction  in  relation  to  the 
aptitudes  presented  by  the  students . 

(5)  the  attitudes  of  the  instructors  toward  the  computer-based  materials. 

(6)  the  preinstruction  skills  and  abilities  brought  to  the  program  by  the 
sample  batch  of  students. 

e.  A sound  long-run  strategy  for  the  evaluation  of  computer  based  instruction 
is  probably  the  method  of  limits . In  this  method,  representative  computer  based 
courses  are  repeatedly  recycled  after  collecting  formative  evaluation  information 
at  the  end  of  each  cycle.  When  successive  revision  cycles  stop  yielding  gains  in 
either  achievement  or  reduced  training  time,  then  the  evaluative  process  should  be 
turned  to  questions  of  cost  effectiveness . The  base  line  for  comparing  the  perform- 
ance of  computer  mediated  training  then  becomes  the  highest  quality  of  hardware, 
software,  and  curriculum  that  can  reasonably  be  achieved.  Cost  trade-offs  involved 
in  opting  for  cheaper  hardware,  fewer  displays,  thinner  programming,  less  "on- 
line" time,  etc.  can  then  be  studied  analytically  by  comparing  the  cheaper  abbre- 
viated version  with  the  maximally  effective  version.  Characteristically,  most 
summative  evaluation  is  upside-down  because  the  comparisons  of  an  experimental 
method  (e.g. , CAI)  are  being  made  against  some  poorly  defined,  nonoptimized 
traditional  method  of  instruction.  Summative  comparisons  of  the  "horse-race" 
variety  also  suffer  from  the  fact  that  costs  of  traditional  programs  have  seldom 
been  determined  in  advance  and  many  of  their  actual  costs  are  hidden  in  other 
categories . 
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{ to  this  connection  the  Army  would  be  well  advised  to  establish  a small  evalu- 
ation unit  on  computer  based  training  that  would  look  beyond  the  Impending  go-no- 
go"  decision  for  meeting  Signal  Corps  training  needs  with  CTS.  Such  • untt  rt~Jd 
be  given  the  resources  on  a small  scale  to  push  the  limits  on  instructional  Quality 
using  a variety  of  criteria.  The  present  evaluation  effort  for  CTS,  aithough  a good 
one  given  the  constraints  of  time  and  resources,  is  dealing  with  limited  hardware, 
limited  operating  system,  and  a limited  concept  of  the  role  of  the  computer  in 

training. 

Responses  to  Selected  Questions 

1 . What  type  of  major  computer  system  for  instruction  is  going 
to  best  fit  the  needs  of  the  Army  by  1980? 

a.  This  is  an  extremely  difficult  question  to  answer  because  of  the  lack  of 
history  of  using  computers  for  instruction.  Modern  computing  is  only  about  25 
Teis  old,  and  it  is  only  in  the  last  15  years  that  any  significant  attentio ,n  has  been 
turned  to  the  use  of  computers  for  instruction.  At  the  moment,  I think  the  nod  from 
the  field  is  going  to  the  dedicated  minicomputer  either  singly  or  in  cl^ter®  * de 
pending  for  the  most  part  upon  the  size  of  the  student  body  to  be  served.  My  own 
appr £tis ai° is  that  the  system  for  the  year  2000  is  the  large-scale  dedicated  processor 
Jith  500  to  1000  terminals.  This  guess  is  based  on  the  anticipation  of  some  new 
two-way  communications  links  that  will  replace  conventional  telephone  lines. 
best  candidate  for  such  a replacement  is  Cable  TV,  where  there  appears  to  be  an- 
cient circuitry  to  handle  250  interactive  computer  terminals  on  a single  TV  chann  . 
There  also  seems  to  be  plenty  of  room  for  computer-dedicated  channels  in  the 
spectrum  of  TV  wavelengths.  The  other  assumption  which  leads  me  to  believe  that 
bj  systems  will  eventually  win  out  over  small  systems  has  to  do  with  thechang^g 
nature  of  pedagogy  for  training  and  instruction.  As  it  is  presently  configured  most 
computerised  instruction,  whether  CAI  or  CMI,  is  frame-orieu *ed  Q^°“n| 
followed  by  simple  feedback  to  the  student.  It  seems  to  me  that  computer  based 
education  of  the  future  la  going  to  have  to  provide  for  muc^ 

instruction  in  which  'he  student  can,  with  greater  independence  than  he  presently 
eniovs  query  a large  data  base  of  information  relative  to  specific  subjects  of  in 
struction  If  true,  then  operating  systems  of  the  future  will  have  to  provide  or 
sonhisticated  retrieval  mechanisms  from  large  complex  data  bases.  Present  con- 

just  don’t  have  enough  ’’muscle”  to  handle  this  changed 

concept  of  instructional  method. 

b.  Another  factor  which  offers  some  food  for  thought  and  could 
requirements  is  the  notion  of  individualized  objectives  for  different  students . TWfl 
concept  will  probably  be  implemented  in  civilian  colleges  and  prep  schools  before  it 
reaches  the  Army’s  training  command,  but  it  is  not  unthinkable  at  the  r levels 

of  training  in  the  armed  forces  to  have  different  sets  of  objectives  for  different 
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capabilities  with  rapid  retrieval.  Individualization  at  the  level  of  objectives  would 
increase  the  requirements  for  sophisticated  diagnosis. 

c.  A third  factor  arguing  for  large  computer  based  systems  is  the  need  for 
uniformity  in  a highly  mobile  world.  Trainees  often  shift  from  one  location  to 
another  and  it  is  wasteful  of  training  time  to  have  students  repeat  segments  of 
training  courses  because  of  minor  variations  in  courses  offered  at  different  loca- 
tions. It  is  much  easier  to  modify  and  update  a single  master  version  of  a course 
of  study  on  a large  system  than  it  is  to  modify  slightly  the  same  course  at  20 
different  locations. 

d.  A fourth  press  toward  large  regional  systems  comes  from  a growth  factor. 

At  present  time,  CAI  and  CMI  development  are  focused  on  individual  courses.  After 
some  growth,  assuming  continued  success  in  performance,  our  concern  will  be  with 
programs  of  instruction  involving  higher  and  higher  levels  of  instruction  for  the 
soldier /student.  When  we  get  to  the  instruction  program  management  level  we  will 
necessarily  want  to  collate  pedagogical  information  about  the  student  from  different 
courses.  The  advantages  of  large  systems  with  resident  data  bases  of  student  records 
from  learning  histories  in  multiple  courses  are  obvious . 

2.  What  new  terminal  hardware  is  on  the  horizon  that  could  open 
up  training  opportunities  ? 

a.  Unfortunately  this  question  is  all  bound  up  with  costs.  Whereas  major  de- 
creases in  cost  for  central  processors  have  resulted  from  breakthroughs  in  tech- 
nology, no  such  impressive  cost  reductions  are  apparently  in  the  offing  for  terminal 
gear.  A complete  student  station  for  an  IBM  1500  system  (circa  1968)  costs  about 
$10,500.  A complete  student  station  for  the  newer  PLATO  system  (circa  1972), 
based  on  a new  plasma  display  technology,  costs  about  $9,  000. 

b.  I believe  there  is  a bright  future  in  training  and  education  for  special  simu- 
lation apparatus,  but  that  future  is  going  to  be  deferred  for  at  least  two  decades. 

3 . What  problems  can  be  expected  if  existing  (planned)  courseware 
wer  ' to  be  converted  for  use  on  the  next  generation  system  (beyond 
ABACUS)? 


There  won’t  be  serious  problems  as  long  as  the  expectations  for  CTS  remain 
with  a frame -oriented  pedagogy  in  the  case  of  CAI  or  with  a CMI  "off-line”  study 
concept.  In  my  opinion,  the  next  generation  of  support  hardware  should  clearly 
be  constructed  with  a different  kind  of  pedagogy  in  the  offing.  Not  much  is  known 
today  about  the  requirements  of  a computer -mediated  inquiry  system  of  instruction. 
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4.  How  can  courseware  development  be  organized  so  as  to  increase 
timeliness  and/or  reduce  costs  without  reducing  quality? 

Courseware  development  does  seem  to  be  the  major  "hang-up"  of  contemporary 
computer  based  systems.  The  TICCIT  effort  has  approached  the  problem  by 
narrowing  its  pedagogical  strategies  to  a "rule -example"  application.  In  spite 
of  this  efficiency-producing  strategy,  its  courseware  production  has  been  badly 
delayed.  The  lack  of  sizeable  blocks  of  integrated  courseware  has  been  seen  at 
the  major  weakness  of  the  PLATO  effort.  Both  TICCIT  and  PLATO  are  being 
funded  by  the  National  Science  Foundation. 

5.  To  what  extent  must  hardware /software/courseware  be  standardized 
to  achieve  transportability  of  courseware? 

It  may  not  be  obvious  to  executive  decision-makers  in  the  Army  that  the  key  to 
this  problem  is  hardware.  If  CAI  terminals  at  the  several  separate  training  in- 
stallations are  maximized  with  respect  to  pedagogical  functions  (e.g. , include  audio 
display),  then  "downward"  compatibility  will  be  almost  automatically  ichieved 
(i.e. , courses  constructed  without  audio  displays  can  be  accommodated  at  ter- 
minals with  audio  capability).  However,  the  reverse  situation  presents  an  im- 
possible problem.  Terminals  are  almost  never  "upward"  compatible  (e.g.,  you 
can't  "run"  an  audio -imbedded  course  on  a terminal  that  has  no  audio  display  de- 
vice). The  latter  situation  most  often  arises  from  misguided  attempts  to  save 
money  on  terminal  gear  by  installing  only  lean  function  equipment.  Civilian  ex- 
perience with  the  transportability  problem  shows  without  doubt  that  it  is  serious, 
and  careful  planning  is  needed  to  avoid  its  bad  effects  on  computerized  training. 

6.  To  what  extent  would  the  introduction  of  CMI  (in  Army  training) 
enhance  learning? 

a.  A program  of  computer  managed  instruction  (CMI)  has  a big  initial  advantage 
over  traditional  instruction  because  the  method  forces  course  authors  to  be  clear 
and  explicit  about  their  objectives.  Modularized  lessons  from  that  point  on  offer 
some  small  advantage  because  of  the  self-pacing  function.  Generally,  however, 

CMI  courses  are  not  very  good  at  meeting  the  needs  of  individual  learners.  Materials 
of  instruction  used  in  CMI  settings  ("off-line")  are  by  and  large  the  same  for  every- 
one, and  it  is  a mistake  to  expect  great  achievement  gains  from  CMI  in  contrast 
with  traditional  instruction  merely  on  the  basis  of  the  fact  that  a computer  is  involved 
in  record-keeping  and  feedback  to  the  student. 


To  my  knowledge  no  one  has  built  a system  of  this  type  with  a reasonably  adequate 
data  base.  There  is,  however,  reason  to  believe  that  they  will  be  built  in  the  future. 
If  inquiry  systems  can  be  made  to  function  effectively,  then  they  hold  great  promise 
of  extending  Man's  powers  of  reasoning,  problem-solving,  estimating,  and  antici- 
pating. 


b.  It  is  in  this  connection  that  a significant  advance  in  training  outcomes  can 
be  made  by  developing  an  appropriate  mix  between  CMI  and  CAI.  For  some  in- 
efficient learners  (poor  study  skills,  inadequate  reading  ability,  and  minimal  moti- 
vation), materials  employing  CAI  formats  may  be  a solution  for  the  chronic  "wash- 
out" or  program  repeater.  CAI  offers  considerable  advantage  if  sufficient  diagnosis 
and  remediation  can  be  built  into  the  "on-line"  materials.  Depending  upon  the  goals 
of  the  student  body,  the  courses  in  the  CTS  program,  and  the  range  of  aptitudes  of 
the  students,  I would  think  that  an  appropriate  mix  of  CMI  and  CAI  would  yield  a 
good  return  on  the  investment.  Where  time  in  a course  is  a premium  criterion, 
the  ablest  students  can  probably  achieve  satisfactorily  with  CMI  materials.  Such 
a plan  saves  time -on -terminal  for  the  less  able  student  in  CAI  mode. 

Recommendations  for  Future  CTS  Long-Range  Developments 

a.  It  seems  to  me  that  the  Army  is  in  a good  position  to  improve  the  quality  of 
its  enlistments  by  using  computerized  instruction  rather  widely  in  its  training  pro- 
gram. An  integrated  Army -wide  computer  based  training  program  could  be  aug- 
mented by  offering  recruits  an  opportunity  to  complete  the  first  two  years  of  college 
(60  semester  hours).  The  rapid  growth  of  junior  and  community  colleges  during  the 
last  two  decades  attests  to  the  expanding  interest  of  young  adults  in  postsecondary 
education. 

b.  As  I envision  it,  approximately  20  college -level  courses  on  such  subjects 
as  history,  sociology,  philosophy,  mathematics,  physics,  chemistry,  biology, 
psychology,  business  statistics,  accounting  and  English  could  be  offered  to  recruits 
during  off-duty  hours  at  training  installations  and  in  operational  units  by  means  of 
an  Army-operated  CAI  system.  The  program  would  require  curriculum  develop- 
ment by  a consortium  of  major  universities  geographically  spread  across  the  country. 
If  their  faculties  developed  the  courses,  then  the  members  of  the  collegiate  con- 
sortium should  have  no  qualms  about  granting  academic  credit  to  soldier/students 

at  the  close  of  their  enlistments  or  of  accepting  such  students  into  upper-division 
programs . 

c.  Working  at  a normal  pace,  students  could  be  expected  to  complete  a 3- 
credit  lower -division  course  in  30  to  40  clock  hours.  This  means  that  20  courses 
could  be  completed  in  600-800  clock  hours.  If  a two-year  hitch  had  at  least  60-80 
productive  weeks,  then  a soldier/student  would  require  only  about  10  hours  per 
week  of  his  off-duty  time  to  complete  freshman  and  sophomore  years  of  collegiate 
instruction.  Of  course,  not  everything  in  college  can  be  learned  on  a computer, 
but  there  would  seem  to  be  no  hesitancy  to  grant  civilian  physical  education  credits 
for  the  supervised  physical  fitness  program  characteristic  of  Army  training. 

d.  This  program  would  attract  the  able  but  indigent  student  who  has  been  reluc- 
tant to  join  the  Army  because  it  would  set  back  his  plans  to  earn  the  baccalaureate 
degree.  The  program  would  also  provide  an  automatic  screening  for  future  officer 
candidate  school  classes . 
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e.  The  investment  required  to  mount  de  novo  this  plan  at  U.  S.  Army  installa- 
tions at  home  and  abroad  would  be  sizeable.  However,  if  the  Army  is  going  to  com- 
puterize its  own  training  program,  it  would  require  relatively  little  additional  sys- 
tems capacity  to  add  on  the  first  two  years  of  collegiate  level  instruction.  With 
success  in  the  lower -division  college  courses,  it  should  be  a relatively  simple 
step  to  the  integration  of  other  types  of  postsecondary  vocational  training  into  a 
joint  Army/Civilian  program  of  instruction. 
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Task  Order  75-63 


Recommendations  Relative  to  Medium-and  Long-Range  Plans  to  Improve 
the  Army's  Computer  Training  System 

by 

Catherine  E.  Morgan 
January  1975 

The  Army  has  an  extensive  and  continuous  training  program  and  is  investigating 
the  optimal  manner  of  incorporating  computer  and  instructional  technology  into  its 
educational  efforts.  To  meet  this  objective  a report  is  to  be  prepared  which  re- 
flects the  experiences  and  opinions  of  civilian,  Army  and  other  government  special- 
ists from  the  field  of  computer  managed  and  computer  assisted  instruction.  This 
report  will  cover  bothshort- and  long-term  plans  for  improving  the  development, 
use  and  evaluation  of  the  computer  based  instructional  systems  in  the  Army. 

Background 

a.  Project  ABACUS,  having  as  its  background  the  Fort  Monmouth  experiences 
in  CAI,  has  transferred  its  operation  to  Fort  Gordon,  Georgia.  The  conference 
held  at  Fort  Gordon  on  December  7-8,  1974  provided  to  the  participants  the  updated 
information  on  which  this  report  is  based.  Some  marked  differences  in  the  project 
were  apparent  from  the  on-site  visit.  Among  these  is  the  obvious  change  in  physical 
location  which  involves  concommitant  changes  in  personnel  and  environment,  but 
far  more  important  is  the  change  in  direction  from  CAI  to  CMI  and  the  concentration 
in  three  training  courses  in  which  development  and  use  had  not  been  experienced 
previously. 

b.  The  principal  deterent  to  making  judgments  relative  to  the  curricular 
materials  and  the  hardware  system  to  be  implemented  is  the  lack  of  an  interactive 
mode  due  to  the  current  software  difficulties. 

Assumptions 

a.  The  writer  of  this  report  assumes  that  the  software  problems  will  be  solved 
and  that  the  interactive  network  will  be  operational  during  the  next  few  months. 
However,  CTS  will  not  have  sufficient  time  to  pilot -test  the  materials  with  students, 
make  the  necessary  revisions  and  validate  them  prior  to  the  time  specified  as  the 
end  of  the  project,  ft  is  hoped  that  the  recommendation  made  at  the  on-site  con- 
ference is  adopted — to  delay  the  evaluation  date  by  a large  enough  time  factor  to 
provide  more  meaningful  data  and  experiences  upon  which  to  make  decisions . 
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b.  It  is  further  assumed  after  reading  some  Army  prepared  manuals  relative  to 
training  materials  that  course  development  (not  only  CAI/CMI)  generally  follows  the 
tenets  of  instructional  technology,  including  task  analysis,  writing  objectives  and 
criterion  items,  constructing  hierarchies  and  determining  instructional  strategies. 

c.  Observations  and  discussions  show  that  the  Army,  Navy  and  Air  Force  have 
an  interest  in  CAI/CMI  systems,  that  each  service  is  involved  to  some  degree  and 
that  there  is  a loose  connection  among  them.  It  is  assumed  that  this  connection  is 
fairly  informal  and  that  each  is  not  influenced  to  any  great  extent  by  the  others . 

d.  The  last  assumption  made  by  the  writer  is  that  the  United  States  Ar  my  will 
continue  to  be  involved  in  computer  based  instructional  technology. 

Discussion 

The  experiences  of  the  author  have  been  in  the  field  of  computer  based  instruc- 
tional systems  for  six  years;  first,  as  an  author-developer  of  course  materials  and 
for  the  last  four  years  as  the  director  of  a computer  assisted  instruction  program 
in  a large  public  school  system.  The  strengths  of  this  CAI  program  have  been  in 
the  development  of  curriculum,  training  of  course  authors  and  users,  implementa- 
tion into  the  classroom  and  in  evaluation.  Therefore,  all  of  the  following  discussion 
has  been  restricted  to  these  areas. 

Training  for  Course -Authors  and  Course-Users 

a.  Experience  has  shown  that  the  implementation  of  already  developed  and 
validated  CAl/CMI  courseware  into  the  regular  instructional  process  requires  some 
unique  skills  and  these  can  be  acquired  by  instructors  through  training  courses . 

This  type  of  course  is  distinctly  different  from  that  required  to  train  course- 
developers  and  obviously  requires  considerably  less  time.  Therefore,  each  CAI 
installation  will  need  to  develop  both  types  of  courses . 

b.  In  the  initial  choice  of  individuals  for  training  to  be  CAI/CMI  developers,  the 
most  important  characteristics  should  be  successful  experience  in  teaching  the 
courses  to  the  types  of  students  with  whom  the  new  technology  will  be  used  and  a 
proven  interest  in  the  individualization  of  instruction  based  on  student  needs . It  is 
unrealistic  to  expect  that  all  those  who  are  trained  will  be  successful  as  developers 
of  CAI/CMI  materials.  During  an  intensive  training  course  which  would  include  all 
aspects  of  design  and  development,  each  participant  should  author  a modular  instruc- 
tional package  which  would  be  critiqued  on  each  area  of  instructional  technology 
following  the  presentation  of  the  topic  in  class.  Like  most  first  efforts  in  any  new 
field,  it  is  unlikely  that  this  product  would  ever  be  used,  but  an  examination  should 
reveal  whether  the  person  has  some  talent  for  the  field. 
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c . With  the  realization  that  the  Army  has  unique  training  requirements  and 
wishes  to  develop  a particular  philosophy  with  its  students,  the  objectives  of  all 
courses  will  have  some  common  denominators . With  this  in  mind,  the  Army  may 
wish  to  develop  and  run  its  own  training  courses  for  authors  and  users  in  spite  of 
some  excellent  university  offerings.  The  Army  has,  in  addition,  another  deterrent 
to  true  individualization  — the  rigid  limitation  on  maximum  course  time.  This 
timetable  concept  may  be  incompatible  with  the  specification  that  solders  may  now 
pursue  the  occupational  specialty  of  their  choice. 

d.  Characteristics  of  superior  author-developers  should  include  mastery  of 
course  design  objectives  which  require  skill  in  logic,  test  construction  and  flow- 
charting and,  in  addition,  in  development  procedures  which  require  creativity  and 
writing  skills.  Individuals  who  possess  this  expertise  should  also  be  able  to  work 
a member  of  a design  team  and  to  work  independently  as  the  situation  dictates.  It 
is  essential  that  developers  have  sufficient  time  and  experience  to  pilot -test  and 
validate  their  instructional  materials.  In  addition,  organizations  should  run  training 
courses  on  a fairly  regular  basis  so  there  exists  a cadre  of  trained  people  from 
which  new  authors  can  be  drawn. 

e.  The  development  of  a shorter  training  course  for  users  of  CAI/CMI  materials 
is  essential.  These  individuals  need  to  be  informed  about  their  roles  in  a changed 
environment.  To  fill  this  role,  they  must  understand  how  the  curriculum  was 
developed  and  be  provided  with  complete  written  documentation  relative  to  the 
objectives,  assessment  items,  hierarchies  and  strategies  of  each  course . Further, 
they  should  interact  with  the  computerized  instructional  programs  at  the  computer 
terminals  in  the  same  way  as  their  students  will . Although  the  skills  required  are 
not  radically  different  from  those  needed  by  an  innovative  classroom  teacher,  in- 
structors in  the  new  environment  will  be  receiving  a plethora  of  up-to-date  informa- 
tion of  a diagnostic  nature.  Their  new  role  will  include  reading  and  interpreting 
this  computer  feedback,  further  diagnosing  student  needs  and  providing  direction 

or  remedial  help  where  necessary.  The  scheduling  of  student  time  will  also  be  the 
responsibility  of  the  instructor  if  the  computer  management  program  does  not  pro- 
vide for  this  function. 

CAI/CMI  Course  Development 

a.  The  first  step  in  course  development  should  be  the  determination  of  need. 

In  the  case  of  computer  based  instruction  this  is  essential,  as  need  here  means 
not  only  is  the  instruction  necessary  but  that  other  methods  will  not  work.  The 
Army  should  conduct  research,  study  its  reco.*ds,  obtain  questionnaires  and  dis- 
cuss with  those  in  the  field  to  determine  the  areas  in  which: 

(1)  The  largest  number  of  deficiencies  are  documented  or  observed 
following  training; 
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(2)  The  most  extensive  need  for  re-education  at  some  later  date  is 
recorded; 

(3)  A number  of  indivuduals  need  a particular  skill  and  that  skill  is  required 
in  widely  separated  geographical  areas— the  number  is  not  large  enough 
to  conduct  a class;  and 

(4)  Course  revisions  of  a somewhat  limited  character  need  to  be  made  fre- 
quently. 

b.  Only  after  this  data  is  obtained  and  analyzed  showing  that  the  present  methods 
are  ineffective,  inefficient  and/or  costly  should  CAI/CMI  be  considered.  As  cost  is 
considered  in  any  course,  the  simplicity  of  revision  in  CAI/CMI  courses  is  a tre- 
mendous asset. 

c.  As  any  organization  who  has  attempted  individual  instruction  has  discovered, 
the  Army  has  determined  that  the  most  diff*  ult  aspect  of  individualization  is  the 
assessing  and  monitoring  student  progress  through  the  courses.  It  is,  therefore, 
obvious  that  a generally  applicable  computer  managed  instructional  paradigm  should 
be  developed  and  tested  following  the  conclusion  of  the  ABACUS  Project  and  inte- 
grating the  positive  features  of  that  model.  From  the  records  of  students  enrolled 
in  such  a CMI  mode,  information  on  specific  individual  and  group  needs  will  be  ob- 
tained. This  data  should  be  carefully  examined  and  the  recurring  trouble  areas  in 
the  courses  be  identified.  Each  trouble  area  should  be  carefully  analyzed  and  the 
determination  of  a solution  should  entail  a close  look  at  the  possibility  of  develop- 
ing a CAI  module.  This  development  effort  could,  in  turn,  become  one  of  the  pre- 
scriptions in  the  overall  CMI  system. 

d.  The  author's  experience  has  shown  that  it  is  impractical  and  indefensible  to 
develop  entire  courses  in  CMI  mode.  Every  course  requires  a variety  of  types  of 
presentations,  i.e.  reading  materials,  group  discussions,  motivational  films, 
lectures  and  practical  hands-on  experiences.  To  assume  that  these  can  all  be 
replaced  by  computer  presentation  is  unrealistic.  However,  drill  materials  with 
immediate  feedback,  interactive  tutorials,  diagnostic  testing  and  simulations  are 
excellent  vehicles  for  individual  learning  and  tying  these  CAI  techniques  to  need 
situations  solves  some  very  sticky  instructional  problems . Another  plus  for  the 
modular  approach  in  the  public  schools  has  been  the  utilization  of  the  materials 
ove:  a number  of  grades  and  in  a number  of  subject  areas . It  would  appear  that 
this  same  situation  occurs  in  Army  training  courses . A student  may  need  to  master 
certain  basics  in  arithmetic,  algebra  and  trigonometry  for  many  courses.  If  CAI 
modules  exist  or  are  developed,  they  can  obviously  be  used  as  segments  of  many 
courses. 
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e.  Army  trainees  are  for  the  most  part  former  public  school  students.  As 
such,  they  reflect  both  the  successes  and  failures  of  public  school  education. 

These  varying  results  are  most  striking  in  the  areas  of  reading  comprehension 
and  vocabulary  which,  in  turn  affect  all  future  training  and  education. 

f.  In  the  preparation  of  materials,  one  Army  author -developer  stated  that  he 
used  the  same  language  in  the  textual  displays  as  that  spoken  in  the  classroom. 

This  approach  is  satisfactory  for  excellent  readers  as  they  comprehend  more  com- 
plex and  technical  written  materials  than  they  could  possibly  attend  to  in  oral  com- 
munication. However  with  less  able  readers,  the  opposite  situation  exists.  These 
students  cannot  comprehend  the  written  language  at  the  same  level  of  complexity 
as  they  can  the  oral  language. 

g.  Therefore  curricular  materials  which  are  to  be  read  both  on  the  visual  com- 
puter display  and  in  printed  form  should  be  analyzed  for  readability  levels.  The 

(analysis  should  include  scores  for  both  comprehension  and  vocabulary  difficulty. 

The  essential  course  vocabulary  will  obviously  be  eliminated  from  any  readability 
analysis  as  these  new  technical  and  other  related  words  and  expressions  are  inte- 
gral to  the  course  and  will  be  taught  prior  to  their  use.  There  is  no  intention  to 
water  down  the  course  content  but  to  look  realistically  at  the  students'  past  achieve- 
ment in  reading.  A student's  instructional  reading  level  is  the  level  in  which  new 
reading  skills  are  acquired,  while  his  independent  reading  level  is  the  one  in  which 
he  uses  previously  acquired  skills.  Optimal  instructional  situations  in  other  sub- 
jects than  reading  make  provision  so  that  textual  materials  are  available  at  the 
student's  independent  reading  level.  If  the  Army  accepts  trainees  whose  achieve- 
ment in  reading  is  below  the  tenth  grade,  it  needs  to  provide  instruction  in  special- 
ties in  a way  that  students  have  a realistic  chance  for  success.  It  is  important  in 
this  connection  to  examine  and  rate  all  Army  training  manuals  related  to  the  courses, 
especially  those  used  in  the  field  for  updating  or  retraining  purposes.  In  addition, 
the  Army  should  prepare  or  acquire  instructional  materials  and  systems  to  teach 
and  improve  reading  achievement. 

Evaluation 

The  evaluation  plan  for  a computer  based  instructional  system  and  network  will 
encompass  its  instructional,  technical  and  cost  features.  However,  the  ultimate 
decision  related  to  instruction  should  be  made  on  the  effectiveness  and  efficiency 
of  learning  which  implies  optimum  technical  support . Cost  figures  on  a research 
and  development  project  do  not  reflect  the  generalizability  of  curriculum  use  for 
other  populations  which  may  ultimately  make  the  system  cost -affordable.  There- 
fore, prime  emphasis  in  Project  ABACUS  should  be  placed  on  the  aspects  of  educa- 
tional evaluation.  The  curriculum  materials  should  be  judged  on  their  validity, 
their  efficiency,  their  acceptability  and  their  effectiveness.  In  addition,  full  credit 
for  course  improvement  and  sequence  validity  should  be  given  to  the  system. 
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The  following  recommendations  are  made  with  the  objective  of  aiding  the  Army 
in  making  medium  and  long-range  plans  for  its  Computer  Training  System: 

a.  The  Army  should  work  closely  with  the  other  two  services  in  all  aspects 
of  computer  based  instruction,  and  particularly  in  the  preparation  of 
curriculum  and  the  specifications  for  future  hardware  and  software.  A 
concerted  effort  should  be  made  to  identify  common  needs,  share  curricu- 
lum and  conduct  comparative  studies  on  the  shared  materials . 

b.  The  Army  should  establish  a central  site  for  curriculum  development.  At 
this  location,  courses  should  be  held  at  regular  scheduled  intervals  for 
both  author -developers  and  instructor-users  of  computer  based  systems. 
From  the  resultant  cadre  of  trained  individuals,  every  effort  should  be 
made  to  retain  the  services  of  effective  course -authors  and  prevent  trans- 
fers to  other  unrelated  Army  assignments. 

c.  The  Army  should  score  all  textual  materials  for  readability  level.  These 
readability  scales  would  not  include  new  technical  works  and  expressions 
essential  to  learning  the  specialty.  This  analysis  would  obviously  use  one 
of  the  recognized  scaling  devices  which  takes  into  account  length  and  com- 
plexity of  sentences. 

d.  The  Army  should  prepare  a generally  applicable  model  for  a computer 
managed  instructional  system.  Inputs  should  include  results  from  Project 
ABACUS  and  from  successful  programs  elsewhere  throughout  the  country. 

e.  The  Army  should  begin  now  to  conduct  a number  of  long-range  research 
projects  which  will  provide  essential  information  relative  to  retention, 
need  for  retraining  and  efficiency  in  the  field.  Questionnaires  and  infor- 
mation sheets  should  be  disseminated  to  National  Guard  and  ROTC  units 
to  determine  their  needs  for  training  and  how  CAI/CMI  can  assist  them  in 
meeting  these  needs. 

f.  As  new  evaluation  plans  are  formulated,  the  Army  should  evaluate  each 
CAI/CMI  effort  with  a detailed  profile  on  a group  of  randomly  selected 
students  to  ascertain  the  correlation  of  success  with  reading  level,  apti- 
tude and  achievement,  including  in  each  course,  a small  number  of  stu- 
dents who  can  join  the  Army  personnel  pipeline  on  a flexible  basis. 

g.  The  Army  should  develop  at  least  one  individualized  course  in  which  indi- 
vidual learning  styles  are  considered.  This  effort  might  determine  that 
for  some  students  a more  traditional  approach  with  its  related  cheaper  cost 
is  most  efficient. 
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h.  The  Army  should  prepare  and  make  available  CAI/CMI  courses  for  field 
retraining,  ROTC  and  the  National  Guard. 

i.  The  Army  should  determine  where  terminals  are  available  in  public  and 
private  institutions  which  can  be  leased  evenings,  weekends,  and  during 
school  vacations . Then  these  terminals  could  be  made  available  for  ROTC , 
National  Guard  and  the  regular  Army  in  cities  across  the  country  and 
abroad. 

j.  The  Army  should  plan  to  place  terminals  in  strategic  locations  within  Army 
installations  to  curtail  travel  time  and  the  obvious  related  subsistence 
allowances  required  when  personnel  are  away  from  their  home  base . 

k.  Working  closely  with  education,  governmental  and  military  institutions, 
the  Army  should  begin  to  set  the  groundwork  for  the  eventual  standardi- 
zation of  one  or  more  instructional  languages. 

Conclusions 


a.  The  Army  continues  to  make  a substantial  contribution  to  the  field  of  com- 
puter based  instruction.  Project  ABACUS  is  a prime  example  of  an  effort  which 
encompasses  all  aspects  of  the  field  from  development  of  specification  for  equip- 
ment and  language  through  the  evaluation  phase. 

b.  The  commonality  of  Army,  Navy  and  Air  Force  requirements  would  suggest 
that  the  three  services  work  closely  on  CAI/CMI  course  development,  share  ma- 
terials and  evaluate  each  other's  products.  Not  only  could  sharing  be  done  among 
the  services  but  also  in  situations  where  it  has  been  determined  to  be  mutually 
advantageous,  with  public  and  private  educational  institutions.  It  would  appear 
that  the  sharing  of  curriculum  by  many  organizations  would  enhance  the  state-of- 
the-art  in  spite  of  the  present  hardware  and  software  incompatabilities . 


c.  Although  the  expansion  of  CAI/CMI  systems  has  not  moved  as  quickly  as  was 
predicted  during  the  1960's,  interest  in  the  mid-seventies  is  at  an  all-time  high. 
There  is  a growing  awareness  that  this  technology  may  be  the  only  way  to  solve 
certain  educational  problems.  The  cost  factor  has  not  decreased  as  rapidly  as 
had  been  hoped.  However,  with  the  high  interest  in  development  and  implemen- 
tation of  curriculum  and  the  new  technologies  on  the  horizon,  the  author  predicts 
that  costs  will  dramatically  decrease  and  by  the  mid-eighties,  large  numbers  of 
computer  terminals  will  be  found  in  all  educational  institutions  and  in  many  private 
homes.  Therefore,  it  is  essential  that  those  in  the  field  continue  to  struggle  and 
proselytize.  Without  these  continuing  experiences,  the  Army  will  not  have  the  re- 
quired expertise  to  Utilize  the  technology. 
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REPORT  NO.  1 

The  Transfer  From  Lab  to  Field  of  Training  Research  Results 

by 

Mr.  Thomas  O'Sullivan 

a.  A key  problem  in  training  is  the  effective  use  in  the  field  of  the  results  of 
training  research.  The  problem  is  of  concern  to  congressional  staff,  OMB, 
DDR&E,  and  each  of  the  Services  (in  both  the  research  and  training  commands). 

b.  The  concern  is  reflected  not  only  in  discussions  but  also  in  the  focus  of 
special  study  panels,  and  the  establishment  of  procedures  to  document  "transfer" 
history. 

c.  The  Services  have  been  providing  mechanisms  to  facilitate  the  transfer 
of  advanced  training  technology  to  the  commands.  In  the  Navy,  the  Navy  Instruc- 
tional Technology  Development  Center,  a CNET  organization  in  San  Diego,  has 
been  working  closely  with  several  of  the  Navy  schools  to  introduce  some  recent 
advances  in  training  technology  into  new  curriculum.  At  Fort  Monmouth,  staff 
members  of  Project  ABACUS,  of  which  the  Computerized  Training  System  (CTS) 
is  a part,  have  been  searching  for  recent  advances  that  will  improve  training  in 
the  Army. 

d.  There  is  a class  of  research  being  performed  in  and  for  the  Service 
laboratories  dealing  with  the  pedagogical  issues  in  training  important  in  the 
context  of  curriculum  design  decisions . This  type  of  research  deals  with  issues 
of:  what  strategy  to  use  given  the  type  course  and  nature  of  the  student  population; 
criterion  for  selecting  and  mixing  presentation  media;  appropriate  instruments 
for  individualization  of  instruction;  definition  of  data  to  be  collected  to  support 
decisions  on  redesigning  or  modifying  the  course,  etc.  Transfer  mechanism  for 
this  class  of  research  seem  weak. 

e.  Discussions  about  the  problem  suggest  that  failure  to  use  the  research 
results  may  be  attributable  to  a number  of  reasons . Often  the  research  results 
are  ambigious.  The  less  ambigious  results  are  generally  demonstrat  d in  a 
tightly  controlled  experiment  with  specified  assumptions  and  regulated  conditions, 
not  within  the  context  of  a total  training  environment.  Researchers  have  difficulty 
pushing  the  concepts  into  the  field  and  must  settle  for  "publishing"  the  results. 
Some  curriculum  implementors  feel  the  research  speaks  to  the  wrong  problems 
but  many,  faced  with  delivery  deadlines,  say  they  fail  to  use  new  techniques  be- 
cause of  lack  of  knowledge  or  time,  and  stick  with  the  older  "tried"  methods  with 
w hich  they  feel  comfortable.  In  effect,  the  new  concept  has  not  been  field  tested 
ind  proven  in  a full  training  environment.  Training  command  personnel  are  not 
familiar  with  the  concepts . There  is  no  formal  mechanism  to  provide  for  such 
tests  on  a regular  basis. 


f.  In  spite  of  this,  many  ad-hoc  attempts  at  transfer  have  been  successful. 

It  seems  reasonable  to  expect  that  if  there  were  a formal  mechanism  to  provide 
for  full  field  test  of  the  laboratory  proven  research  result  then  doubt  would  be 
put  to  rest,  one  way  or  another,  and  more  transfer  would  take  place. 

g.  A lab  school,  providing  for  such  field  tests  in  the  civil  sector  implies  a 
well  staffed,  full  function  school  handling  a body  of  curriculum  with  real  students. 
The  students  may  represent  a particular  set  or  mix  of  socio-economic  background. 
The  school's  purpose  is  usually  for  the  experimental  introduction  of  new  proce- 
dures or  curriculum  or  special  situations  with  observation  and  evaluation  of  the 
effects.  Access  to  the  school  is  provided  to  an  academ.c  department  or  a special 
project  or  program  staff. 

h.  The  military  correlate  to  a lab  school  would  be  similar.  It  implies  location 
at  a training  command  school  with  appropriate  student  flow  and  with  selectively 
developed  exemplary  curriculum  in  a number  of  different  courses  to  provide  an 
advanced  baseline  for  work  with  different  kinds  of  students  and  different  kinds  of 
course  material  (i.e. , different  mixes  of  cognitive,  procedural  and  psychomotor 
skill  content).  Access  to  the  school  and  its  curriculum  and  student  flow  should  be 
provided  to  military  training  research  laboratories  and  their  contractors,  under 
controlled  conditions,  to  perform  systematic  experimentation  on  issues  relevant 

to  curriculum  design  and  implementation. 

i.  In  order  to  support  the  exemplary  cirriculum  and  the  experimentation,  the 
lab  school  must  be  equipped  w'ith  advanced  hardware  including:  computer  managed 
instruction  (CMI),  computer  aided  instruction  (CAI),  special  computer  controlled 
devices  (carrels,  slide/tape,  cassette/film  displays,  TV/tape,  generalised  simu- 
lators, etc.).  In  addition,  special  test  and  measurement  implementation  staff  is 
needed  to  support  the  research  efforts,  and  access  must  be  provided  to  the  re- 
searchers for  purposes  of  curriculum  modification. 


j.  In  the  ARPA/Joint  Services  field  tests  of  the  PLATO  system  at  ,‘dt>erdeen, 
Shepherd,  Chanute,  and  San  Diego,  it  was  hoped  that  the  terminals  at  each  school 
would  be  used  to  establish  laboratories  in  which  research  results  from  the  lab  could 
be  checked  out  in  the  context  of  a full  training  environment.  The  situation  seamed 
ideal.  There  would  be  curriculum  in  place,  and  a flow  of  students  available  for 
systematic  variance  of  presentation  and  selection  of  experimental  and  comrol 
groups.  However,  there  are  no  signs  to  date  that  this  transition  will  take  place. 

k.  The  Air  Force  AIS  s stem  in  development  at  Lowry  has  the  potential  for 
becoming  the  hub  of  an  experimental  or  laboratory  school.  Smc  ; the  primary 
thrust  is  to  provide  Compute^  Managed  Instruction  (CMI)  caprbi  .ity  and  demon- 
strate the  effectiveness  of  CMI  for  different  kinds  of  curriculum,  it  will  have  the 
capability  of  gathering  test  data  on  mixed-media  presentations  of  course  materials. 
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While  there  is  talk  of  putting  some  Computer  Aided  Ir  itruction  (CAI)  material  up 
on  the  AIS  there  is  no  firm  plan  to  do  so,  and  while  there  is  hope,  there  is  no  firm 
indication  that  any  of  the  clusters  of  terminals  will  be  set  up  in  a formal  laboratory 
school  of  the  type  indicated  above. 

l.  The  Navy  researci:  personnel  at  San  Diego  identified  the  requirement  for 
such  a lab  school  and  have  laid  out  detailed  plans  for  its  implementation,  however, 
the  plans  have  not  as  yet  been  approved. 

m.  The  Army  CTS  project  has  the  potential  for  providing  such  a lab  school 
environment,  tt  has  the  CAI/CMI  orientation  needed  for  such  on  operation.  While 

it  is  currently  a single-terminaRype  system,  it  has  the  capability  of  handling  multi- 
ple types  of  student  and  instructor  input  and  output  devices.  Its  structure  allows  for 
easy  access  by  researchers  to  curriculum  for  purposes  of  modification.  Each  of 
these  areas  will  be  discussed. 

n.  The  CTS  system  provides  a student  record  block  with  a majority  of  the 
record  available  for  use  by  the  instructor /designer.  This  fixed  size  record  should 
provide  sufficient  space  to  collect  the  necessary  data  for  most  experiments.  In 
cases  where  this  is  not  true,  it  should  be  possible,  with  little  effort,  to  provide  for 
links  to  "extended'  student  record  fields  in  both  the  terminal  driver  where  the  data 
is  collected  and  in  the  main  processor  where  the  records  are  processed  both  by 
the  system  and  by  application  programs.  There  may  be  situations  in  which  the 
CMI  function  is  substantial  enough  to  require  substantial  computer  capacity  but 
this  is  likely  to  be  between  student  sessions.  If  not,  then  the  experimental  design 
should  be  examined  to  see  if  the  function  can  be  provided  in  a different  way,  or  a 
link  provided  to  a more  powerful  system  (such  as  the  Air  Force  Alb)  in  real  time 
to  complement  the  CTS  capacity. 

o.  The  system  architecture  provides  for  separate  terminal  divers  for  each 
terminal  cluster.  Present  design  calls  for  a single-terminal  type.  However,  with 
a reprogramming  of  the  terminal  drivers  other  devices  could  be  supported.  For 
example,  a rectnt  def  ign  study  supported  by  ARPA  and  executed  by  SOFTEC  of 
Waltham,  Massachusetts,  shows  that  PDP  ll's  can  be  programmed  to  handle  the 
TUTOR  language  author  sub-set  of  code  to  drive  PLATO  terminals.  The  study 
shows  that  the  software  development  will  be  expensive  but  if  the  conventions  of 
lesson  storage  and  student -recoid  data  are  followed,  PLATO  course  materials 
could  be  supported  on  one  terminal  cluster  of  a CTS  system.  Similarly,  other 
terminals  could  be  supported  in  addition  to  specially  designed  computer  driven 
multi-device  student  carrels.  One  terminal  driver  could  be  configured  to  swap 

in  special  software  drivers  from  the  local  disk  along  with  student  lesson  material 
to  accommodate  currently  undefined  student  interface  devices,  both  input  and  out- 
put. 
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p.  CTS  staff  have  been  exploring  the  design  requirements  to  allow  the  system 

to  support  both  remote  terminals,  and  remote  clusters  including  the  terminal  driver. 
If  both  of  these  capabilities  are  implemented,  then  the  flexibility  of  the  system  will 
improve  substantially.  Researchers /authors  will  be  able  to  access  the  author  sup- 
port capability  of  CTS  from  geographically  distributed  sites  permitting  Service 
research  labs  and  their  contractors  to  access  curriculum  materials,  modify  them, 
and  check  them  out  in  preparation  for  an  experiment.  At  a modest  level,  research- 
ers could  perform  lab  based  (as  opposed  to  school  based)  experiments  prior  to 
release  of  the  new  materials  to  the  school. 

q.  The  administrative  organization  of  the  lab  school  presents  some  interesting 
problems.  Certainly  the  activity  could  be  centrally  administered  and  controlled. 

That  is  the  school  where  the  laboratory  is  located  or  a single  organization  there 
could  have  complete  control  of  all  functions.  However,  this  could  tend  to  dampen 
broad  and  flexible  use  of  the  system.  The  functional  distribution  of  responsibility 
and  authority,  carefully  separated,  could  encourage  maximum  use  of  the  facility 
as  a research  resource  and  as  a transfer  device. 

r.  From  a logical  standpoint,  the  administration  of  the  lab  school  system 
breaks  into  three  areas: 

Course  Administration 

Computer  & Device  Center  Administration 

Research  Administration 

s.  It  appears  that  from  a functional  standpoint  this  breakout  is  clean  and 
clusters  like  activities  together  under  separate  specialized  groups. 

t.  Course  administration  is  classically  a school  responsibility  and  under  the 
proposed  lab  school  plan  would  remain  so.  With  respect  to  the  courses  for  which 
exemplary  base  line  curriculum  is  developed,  they  would  be  an  integral  part  of 
the  responsibility  of  that  school's  staff  and  its  commander,  instructor  and  admin- 
istrative staff  would  be  provided  through  noimal  channels.  This  allows  the  school's 
command  to  systematically  expose  personnel  to  the  new  techniques  and  technology 
used  in  the  course.  The  problem  of  scheduling,  base  logistics,  and  billeting  at 
which  schools  have  to  be  proficient  would  remain  with  the  school . It  may  be  that 
the  courses  are  distributed  across  a number  of  schools  in  order  to  get  the  right 
mix  of  type  of  student  and  type  of  course. 

u.  Courses  into  which  advanced  technologies  have  been  incorporating  incor- 
porated have  special  problems.  The  computer,  its  terminals,  communications, 
and  special  devices  require  specialized  knowledge  to  keep  them  available  on  a 
stable  basis.  Their  acquisition,  storage,  operation,  issuances,  maintenance, 
and  service  will  probably  require  the  formation  of  a special  group.  At  least  a 


special  branch  of  an  existing  group  Buch  as  the  computer  center  if  appropriate, 
or  audio  visual  device  center,  etc.  The  course  administrator  should  not  have  to 
cope  with  the  standard  service  groups  on  base,  but  have  access  to  the  specialized 
group  who  have  the  knowledge  and  motivation  necessary  to  provide  direct  service, 
or  the  clout  needed  to  act  as  expeditor  for  the  course  administrators  as  student 
population  changes,  class  locations  are  changed,  equipment  fails,  etc. 


v.  Research  administration  can  be  a third  discrete  component  of  the  operation 
and  is  probably  made  up  principally  of  a lightly  staffed  committee,  ft  will  have  the 
responsibility  of  screening  applications  to  conduct  research  in  the  lab  school, 
select  between  competing  requirements  for  the  lab  resource,  schedule  the  experi- 
ments so  as  to  not  permit  interference  between  proposed  projects,  and  protect  the 
interests  of  the  school  and  its  students.  Thus,  while  the  primary  representation 
on  this  steering  committee  should  be  from  the  Service  research  laboratories  who, 
together  with  their  contractors,  are  making  use  of  the  lab  school,  there  should 

be  representation  from  the  school  command,  and  the  administration  of  the  relevant 
course  who  should  have  virtual  veto  power  to  reject  proposed  experiments  that  seem 
unjustified  or  dangerous  to  the  students.  The  staff  should  support  the  steering 
committee  but  report  to  an  executive  committee  with  representation  from  the 
training  and  research  commands  with  the  authority  to  rule  over  disputed  issues. 

w.  This  paper  has  begged  the  issue  of  course  selection  and  the  development  of 
exemplary  curriculum.  If  the  lab  school  is  to  be  made  available  to  other  services 
for  research  purposes  then  the  selection  and  design  should  be  done  by  a tri-service 
task  group  with  background  in  training  research  in  each  of  the  Services . 

x.  The  establishment  of  a formal  test  bed  in  the  operational  Training  Command 
to  provide  a controlled,  but  real,  training  environment  to  use  in  testing  newly  lab 
proven  techniques  in  a formal  course  with  constant  student  flow,  would  provide 
several  advantages. 

. (1)  Give  a fair  full  test  of  the  research  concept. 

(2)  Introduce  Training  Command  personnel  directly  to  the  techniques. 

(3)  Relieve  the  research  community  of  the  burden  of  running  'lab  schools". 

(4)  Relieve  Training  Commands  from  the  burden  of  providing  test  environ- 
ments on  an  ad-hoc  basis. 


y.  If  the  proposed  lab  school  is  to  be  established,  it  should  be  done  on  an 
evolutionary  basis.  The  basic  CTS  design  seems  to  be  flexible  enough  to  support 
the  activities  described  above.  E must  now  be  proven  that  the  design  works.  In 
selecting  options  for  implementation  under  the  design,  special  care  must  be  taken 
in  the  design  of  the  data  collection  and  reduction  packages  so  that  the  CMI  section 
is  powerful  enough  to  support  a lab  school  operation. 
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REPORT  NO.  2 

The  Delivery  of  Job  Related  Information  to  the  Specialist 

by 

Mr.  Thomas  O'Sullivan 

a.  Formal  school  based  technical  training,  at  the  specialist  entry  level,  is 
the  beginning  of  a series  of  operations  involving  the  delivery  of  job  related  infor- 
mation, in  which  the  specialist  will  be  involved  for  the  rest  of  his  work  career. 
The  series  of  operations,  if  evaluated  and  planned  as  a system  within  the  bound- 
aries of  advanced  technology  can  provide  better  performance,  and  therefore 
readiness,  at  lower  cost.  Thus,  the  Army  CTS  could  prove  to  be  an  effective 
tool  within  the  context  of  formal  school  based  training  and  extend  its  impact  to 
operational  bases. 

b.  Thj3  paper  will  touch  on  some  of  the  key  issues  and  advantages  in  an 
overall  delivery  system  view,  as  defined  above,  in  formal  school  training,  formal 
on  base  training,  on  the  job  training,  the  management  of  daily  task  assignments 
to  reinforce  training,  proficiency  maintenance,  and  job  performance  aids. 

c.  The  potential  for  using  advanced  technology  to  deliver  and  manage  indi- 
vidualized instruction,  and  realize  some  of  the  savings  attendant  to  the  process 

is  well  known.  Much  less  is  known  about  how  to  integrate  the  management  across 
mixed  media  and  forms  of  delivery  that  employ  automated  and  semi -automated 
techniques.  However,  the  Navy  funded  work  at  the  University  of  Illinois  on  the 
CAMUS  project  showed  success  in  managing  students  through  the  study  of  econo- 
mics by  directing  students,  on  an  individualized  basis,  to  various  learning  re- 
sources. When  we  are  able  to  apply  these  techniques  to  courses  where  instructor 
involvement  has  beei  minimized,  we  should  be  able  to  transport  the  technology 
from  the  school  to  the  operational  base,  reducing  the  school  residency  time  re- 
quirements for  the  courses  involved  and  benefit  from  the  attendant  savings.  CTS 
as  a delivery  and  management  system  can  play  an  important  role  in  this  transition. 
Early  experiments  in  the  Air  Force  with  a generalized  simulator,  in  the  Navy  with 
dockside  training,  and  in  the  Army  with  UTEC  II  show  promise  for  moving  in  this 
direction  more  rapidly  by  providing  new  insight  into  the  control  of  the  training 
process  and  new  technology  for  direct  presentation  of  training  materials  and  ex- 
periences . 

d.  The  shift  of  more  training  away  from  the  school  to  the  operational  base 
will  not  permit  a reduction  in  manpower.  There  will  be  more  men  on  base  but 
each  man  will  spend  less  time  on  operational  tasks.  The  base  commander  must 
take  care  to  insure  that  the  formal  training  time  is  not  an  extra  duty  but  a part 
of  the  men's  normal  duty  day.  The  advantage  of  having  the  extra  manpower  is 
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that  in  time  of  emergency  readiness  is  higher,  since  the  men  are  already  inte- 
grated into  the  base  operations.  This  form  of  training  should  not  be  considered 
on  the  job  training.  It  is  formal  school-like  training  and  this  mode  is  similar  to 
work-study  programs  in  the  civil  sector. 

e.  The  course  at  Sheppard  Air  Force  Base  for  Physician's  Assistant  is  a 
two  year  program.  The  first  year  is  spent  as  a resident  student  at  the  school 
with  1000  hours  plus  of  student  contact.  The  student  also  has  ’'hand-on"  experi- 
ence at  local  military  medical  facilities.  The  second  year  is  spent  in  on  the  job 
training  at  the  Air  Force  medical  facility  to  which  the  trainee  is  assigned. 

f.  The  first  year  curriculum  is  currently  being  revised.  A substantial  portion 
is  being  put  up  on  PLATO.  If  the  course  were  put  under  Advanced  Instructional 
System  management  control,  using  CAMUS  techniques  and  the  portions  not  amen- 
able to  CAI  were  presented  through  UTEC  II  and  other  such  devices,  instructor 
load  could  be  minimized.  It  is  possible  that  through  a reorganization  of  the 
ordering  of  the  course  elements  that  a substantial  portion  of  the  first  could  be 
transferred  to  the  station  to  which  the  student  is  to  be  transferred  for  his  second 
year  by  establishing  a learning  center  there  with  the  necessary  equipment  and 
materials.  Lab  experience  at  the  school  could  be  minimized  and  replaced  with 

on  the  job  training  at  the  new  station . If  that  station  does  not  have  the  appropriate 
laboratory,  then  there  is  question  as  to  whether  the  trainee  should  be  given  the 
same  lab  experience  as  those  who  will  be  stationed  at  a location  where  those  lab 
facilities  are  available. 

g.  The  shortening  of  the  school  residency  time  and  lengthening  of  the 
second  period,  changing  it  to  a work -study  program  would  result  in  placing  use- 
ful personnel,  with  a specialty  in  short  supply,  at  operational  facilities  ready  to 
move  into  action  in  cases  of  medical  emergency. 

h.  Although  on  the  job  training  already  takes  place  on  operational  bases 
for  many  MOS's,  the  author  has  seen  little  evidence  that  it  is  being  effectively 
executed,  and  often  its  implementation  varies  broadly  as  a function  of  the  super- 
visors' ability  and  interest.  The  addition  of  CMI  to  present  formal  criterion- 
referenced  testing  and  to  collect  actual  performance  data  could  lead  to  a more 
consistently  high  level  of  achievement  for  students  in  on  the  job  training  pro- 
grams. In  addition,  the  test  and  performance  data  in  the  CMI  system  could  be 
used  to  generate  guidelines  in  the  form  of  preferred  priority  listings  matching 
trainees  to  the  day  to  day  task  assignments  to  make  most  effective  use  of  the 
work  day  in  strengthening  trainees  in  their  weak  areas  of  knowledge  and  per- 
formance and  in  spotting  where  formal  training  or  other  remediation  is  needed. 

i.  For  example,  if  recently  trained  auto  mechanics  are  given  an  on  the 
job  training  assignment,  and  their  day  to  day  assignments  are  placed  under 
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CMI  control,  the  exposure  to  different  kinds  of  experience  could  be  balanced 
across  the  trainee  group.  When  the  level  of  exposure  to  a particular  type  of 
work  experience  drops  below  a prescribed  threshold,  the  trainee  could  be 
tested  in  that  area  to  determine  whether  or  not  he  should  be  exposed  to  a simu- 
lation or  other  drill  and  practice  exercise.  Similarly,  if  the  measured  perform- 
ance of  the  trainee  falls  below  a threshold  for  his  level  of  training  and  experience, 
then  remedial  action  could  be  prescribed. 

j.  The  same  kind  of  test  and  management  system  for  daily  assignments 
should  benefit  the  experienced  specialist  as  well.  Whether  it  is  used  to  system- 
atically expose  the  experienced  man  to  knowledge  and  procedures  to  reinforce 
prior  learning  and  experience,  or  to  prepare  him  for  a specific  new  assignment, 
these  techniques  should  reduce  the  requirement  for  formal  proficiency  mainten- 
ance programs. 

k.  If  the  system  notes  that  an  experienced  electronics  technician  has  not, 
in  his  normal  work  experience,  been  exposed  for  some  time  to  a particular  prob- 
lem or  procedure,  the  task  list  of  outstanding  work  could  be  scanned.  If  that  type 
task  is  found,  the  technician  could  be  placed  on  the  list  of  preferred  workers  for 
that  task.  This  could  be  used  as  a guideline  but  not  acted  on  by  the  supervisor. 
However  if  the  technician  is  bypassed  too  often  then  the  situation  could  be  flagged, 
and  the  situation  examined  to  determine  if  there  is  special  reason  for  lack  of 
response.  When  the  technician  does  perform  against  the  assignment,  measure- 
ment of  the  quality  of  the  work  could  be  measured  to  determine  if  remedial  action 
should  be  prescribed. 

l.  Even  though  reduced,  the  residual  proficiency  maintenance  program  can 
be  improved  at  low  cost,  assuming  that  formal  training  has  been  individualized 
and  is  automated  or  semi -automated.  Because  of  the  individualization,  the  ex- 
posure to  materials  for  proficiency  maintenance  should  be  able  to  be  supported 
by  the  formal  course  materials.  Through  the  testing,  the  specialist  will  be 
guided  through  only  those  materials  he  needs . This  coupled  with  the  daily  task 
scheduling  system  should  provide  adequate  review  for  proficiency  maintenance. 

m.  The  system  of  information  delivery  is  complex  but  lends  itself  to  two 
formal  models  of  the  process  and  the  relationship  between  the  components.  It 
could  be  either  developed  on  a technology  base  across  systems,  or  on  a system 
life  cycle  base  across  technologies. 

n.  The  analysis  for  the  technology  based  model  could  start  by  looking  at 
the  key  or  critical  skills  involved  in  base  maintenance  operations,  and  back  off 
through  the  key  components  of  the  information  system  for  each  base  function  or 
weapon  system. 
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o.  If  the  skill  selected  were  vehicle  maintenance,  then  the  use  of  that 
skill  would  be  examined  for  the  whole  base  or  operational  unit  on  that  base  for 
preventative  maintenance,  field  maintenance,  and  garage  or  shop  for  vehicle 
and  shop  or  depot  for  parts  rebuilding  or  repair.  The  model  should  identify 
the  knowledge  and  ability  required  for  each  operation  to  be  performed  and  the 
normal  progression  of  the  specialists  growth  through  the  system  for  specific 
weapons  systems  and  base  functions. 

p.  The  analysis  for  the  weapons  system  based  model  could  start  with  a 
major  weapons  system,  identifying  the  critical  skills  necessary  through  the  life 
cycle  of  the  system  for  maintenance  and/or  operations , then  back  off  through  the 
key  components  of  information  delivery  for  each  skill  or  technology. 

q.  If  the  system  selected  were  a ground  to  air  missile  system,  the  sub- 
system and  related  technical  skills  would  be  identified  such  as  rau..x  transmitter 
and  receiver,  driving  mechanism,  power  generation,  launcher,  weapon,  vehicle, 
and  communication  and  the  associated  processes  required  for  the  electronic  and 
mechanical  maintenance  (including  preventative,  field,  and  shop  activities).  Then 
for  each  of  the  technical  areas  the  knowledge  and  ability  for  each  operation  would 
be  identified  along  with  the  normal  growth  of  the  specialist  through  the  system. 

r.  The  models  from  each  approach  would  be  expected  to  have  a high  degree 
of  commonality.  Much  of  the  information  required  probably  exists  in  previously 
performed  task  analyses.  In  either  case,  it  should  be  possible  to  use  the  model 
to  identify  new  organizations  of  the  delivery  processes  from  formal  school  based 
training  through  proficiency  maintenance  that  would  reduce  costs  and  improve 
readiness. 

s.  A simplifying  key  lies  in  job  performance  aids.  Effective  performance 
of  the  specialist,  assisted  by  his  JPAs,  is  central  to  reduced  costs  and  increased 
readiness,  and  therefore  the  general  goal  of  technical  training.  Since  20  percent 
of  the  specialist's  time  is  spent  in  training,  optimizing  the  development  and  effec- 
tive use  of  JPAs  for  80  percent  of  the  specialist's  career  time,  and  backing 
through  the  information  delivery  chain  to  define  the  role  of  the  other  components 
has  minimum  risk  of  sub -optimization. 

t.  Computer  based  systems  such  as  CTS  have  the  characteristics  neces- 
sary to  provide  the  management  for  and  direct  control  over  some  of  the  delivery 
of  information  in  all  of  the  above  components  in  an  individualized  mode. 

u.  Thus  as  information  base  (curriculum)  and  associated  management 
system  might  be  designed  for  a whole  technology  or  specialty  with  the  parameter 
for  individualization  to  cover  not  only  the  difference  between  specialists,  but  also 
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their  assignments,  and  the  different  modes  of  use  (from  schooling  through  OJT 
and  proficiency  maintenance  to  JPA).  We  probably  know  enough  today  to  organize 
such  a data  base  and  superimpose  onto  it  different  pointing  structure  and 
branching  mechanisms  to  provide  a system  of  such  scope.  As  our  understanding 
of  artificial  intelligence  grows  we  should  be  able  to  find  even  more  efficient  ways 
of  generating  the  pointing  structure  and  branching  algorithms  to  cover  new  wea- 
pons systems  and  devices  introduced  into  the  arsenal  and  its  related  logistics 
support.  In  some  specialties  individualization  discriminating  between  the  needs 
of  the  active  force  and  the  reserve  force  might  be  accommodated.  In  other 
specialties  where  there  is  sufficient  information  in  common  across  the  Services, 
a single  system  with  service  specific  individualization  parameters  might  be  con- 
sidered. 
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Introduction 


! 


a.  Based  upon  the  experience  of  an  on-site  visit  to  Fort  Gordon,  Georgia, 
the  reading  of  Project  ABACUS  reports  and  manuals  and  conversations  with 
Army  officers  and  enlisted  men,  this  paper  will: 

(1)  Offer  comments  and  suggestions  for  the  current  Project  ABACUS. 
Comments  on  the  current  Project  ABACUS  are  offered  as  alterations  to  an  existing 
"flight  plan".  This  author  feels  that  several  alterations  in  the  current  Project 
ABACUS  environment  must  be  made  for  effective  extrapolation  of  project  findings 
to  Army -wide  Computerized  Training  Systems  (CTS). 

(2)  Make  recommendations  for  the  design  and  implementation  of  future  com- 
puterized training  systems  in  the  Army . Recommendations  for  the  design  and  imple- 
mentation of  future  computerized  training  systems  are  based  upon  the  observations 
and  experiences  of  this  writer;  however,  findings  from  the  current  ABACUS  Project 
must  lend  the  refinement  necessary  for  fruitful  implementation. 

b.  In  reality,  recommendations  for  the  current  project  and  for  the  future  pro- 
ject become  inseparable. 

Suggestions  for  the  Current  ABACUS  Project 


a.  The  existing  preliminary  evaluation  plan  dated  January  19,  1974,  should  be 
updated  and  redistributed  as  soon  as  possible.  Careful  monitoring  of  its  imple- 
mentation should  begin  immediately  thereafter. 

(1)  Given  the  change  in  location  for  the  test  site  from  Ft.  Monmouth,  New 
Jersey,  to  Ft.  Gordon,  Georgia,  and  given  the  change  in  amount  of  CMI  vs  CAI 
materials  used  in  the  project,  the  evaluation  plan  is  in  much  need  of  adjustment. 

(2)  For  example,  the  evaluation  plan  dated  January  19,  1974,  makes  careful 
distinction  between: 

(a)  CAI.  In  this  mode,  the  lesson  material  is  stored  on  line  within  the 
computer  system  and  the  computer  serves  as  the  instructional  medium. 

(b)  CMI.  In  this  mode,  the  computer  provides  lesson  assignments, 
schedules  equipment  and  instructional  media,  scores  tests  and  provides  remedial 

work  and  monitors  student  progress.  All  lesson  material  is  presented  via  off-line 
media. 
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(c)  CPI.  This  mode  may  be  considered  as  a combination  of  CAI  and 
CMI  with  part  of  the  lesson  material  stored  on  line  and  part  off  line.  The  computer 
is  one  of  several  instructional  media  used  within  the  lesson. 1 

(3)  These  definitions  appear  useful  and  helpful;  however,  if  the  delineations 
are  not  maintained  in  course  development  and  project  implementation,  the  evalu- 
ation loses  a valuable  facet.  Project  and  local  staff  at  Ft.  Gordon  must  be  aware 
of  and  consciously  control  the  amount  and  location  of  CAI,  CMI  and  CDI 
materials  present  in  the  current  project.  Without  such  control,  extrapolation  of 
the  ABACUS  Instructional  Design  Model  to  future  courses  and  cost  effectiveness 
studies  for  such  extrapolation  will  be  difficult.  The  cost  of  building  an  airplane 
from  various  prototypes  depends  directly  on  the  component  characteristics  within 
the  prototypes.  Without  detailed  specifications  for  the  engines  used,  for  example, 
intelligent  choice  of  which  to  use  in  future  airplanes  (and  for  which  purpose)  would 
be  impossible. 

(4)  Discussions  at  Ft.  Gordon  in  December,  1974,  indicated  that  conscious 
distinction  among  the  three  types  of  use  of  the  computer  in  the  instructional  process 
is  not  present  in  the  minds  of  the  instructional  developers,  course  instructors  or 
project  staff.  This  observation  alone  seems  to  dictate  the  need  for  an  on-site 
evaluator  to  operationalize  evaluation  implementation  at  Ft . Gordon  and  to  effect 
coordination  with  the  central  ABACUS  staff. 

b.  An  alternative  view  of  the  teaching/learning  process  should  be  considered. 

(1)  In  actuality  it  is  often  very  hard  to  predetermine  where  CAI,  CMI  or 
CDI  will  be  used.  The  application  and  the  teaching  approach  may  help  determine 
which  type  of  computer  assisted  learning  is  best,  but  generally,  a combination  of 
types  is  the  most  effective  teaching  method.  Given  the  difficulty  in  matching  appli- 
cations with  type  of  computer  involvement,  might  it  not  be  more  fruitful  to  define 
components  of  the  teaching/learning  process  and  then  determine  "if"  and  "how  ' 
the  computer  could  be  useful?  For  example,  the  teaching  of  algebraic  concepts 
might  be  broken  down  into  the  following  components : 


(a)  initial  presentation  of  material 

(b)  drill  and  practice  of  the  materials  presented 

(c)  evaluation  of  student  knowledge  of  the  material,  and 

1 

(d)  reteaching  of  the  materials  (if  necessary). 


XA  PRELIMINARY  INSTRUCTIONAL  MODEL  FOR  A COMPUTERIZED  TRAINING 
SYSTEM,  Interim  Report,  July,  1973,  page  5 
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(2)  In  this  example,  initial  presentation  of  materials  might  be  less  effectively- 
offered  by  the  computer  than  by  a teacher,  a text  or  audiovisual  mechanism.  Drill 
and  practice  is  most  probably  best  adapted  to  CAI.  Combination  of  evaluation  and 
reteaching  components  could  be  done  by  the  computer  in  a CAI  orCDI  mode  while 
the  overall  progress  of  a student  through  the  various  modules  in  an  algebraic  course 
and  his  ultimate  testing  could  be  a C MI  function.  Thus,  the  type  and  scope  of  com- 
puter involvement  (if  any)  is  determined  by  the  needs  of  the  teaching/learning  pro- 
cess rather  than  having  the  computer  involvement  superimposed  on  the  teaching/ 
learning  process. 

c.  A student  flow  model  should  be  considered. 

(1)  Development  of  a model  of  the  student  flow  through  the  teaching/lcarning 
process  described  above  might  be  helpful.  This  model  could  be  applied  to  the  teach- 
ingAearning  process  at  each  learning  unit  level  (task,  subtask,  etc.).  An  example 
of  such  a flow  for  the  smallest  unit  of  learning,  designated  as  a task,  subelement,3 
is  shown  in  Figure  1.  This  flow  differs  from  the  general  course  structure  flow  and 
from  the  skill,  practice,  posttest,  task  element  problem  and  lesson  test  strategies 
(as  described  in  the  Preliminary  Instructional  Model)3  in  that  it  depicts  movement 
through  material  from  the  student  perspective  rather  than  from  the  course  structure 
or  instructional  design  perspective. 

(2)  Figure  1 is  taken  from  the  Ohio  State  University  College  of  Medicine 
Independent  Study  Program.3,  4 This  model  would  need,  to  be  altered  to  incorporate 
the  skill  and  performance  testing  of  the  three  ABACUS  training  test  courses.  Still 
the  concept  is  generally  valid  in  that: 

(a)  The  student  reviews  written  objectives.  These  objectives  are 
developed  by  the  faculty  teaching  the  course  and  are  given  to  the  student.  Thus, 
expected  performance  and  knowledge  levels  are  clearly  stated  for  both  the  student 
and  the  instructor. 


2A  PRELIMINARY  INSTRUCTIONAL  MODEL  FOR  A COMPUTERIZED  TRAINING 
SYSTEM,  Interim  Report,  July,  1973,  page3. 

3Griesen,  James  V.,  "Computer  Assisted  Idependent  Study",  Journal  of  Clinical 
Computing,  Vol.  2,  No.  4,  pp.  68-76,  1973. 

INDIVIDUALIZING  THE  STUDY  OF  MEDICINE,  various  authors  of  The  Oh;o  State 
University  College  of  Medicine  Independent  Study  Program,  published  by  Educa- 
tional Products,  Inc.,  5005  W.  HOthSt.,  Oak  Lawn,  Illinois  60453,  available 
January,  1975,  $24.00. 
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Figure  1.  Sample  student  flow  model. 


(b)  Determination  of  the  need  for  study  is  made  by  the  student  yif 
mature  enough)  or  by  pretest  criteria. 


(c)  If  the  student  needs  further  study  then  a variety  of  learnn.^  resources 
are  used.  These  resources  are  delineated  by  faculty  and  include  printed  material, 
models,  audiovisual  aids,  clinical  experiences  and  CAI  learning  modules. 

(d)  If  the  student  does  not  need  study  as  determined  by  pretests  or  by 
his  own  assessment  he  moves  to  a Tutorial  (self)  Evaluation  System  (TES)  where 
the  evaluation  and  reteaching  components  are  combined  via  CAI/CMI.  Students  pro- 
ceed to  use  the  tutorial  (self)  evaluation  system  whenever  they  feel  "ready". 

(e)  If  the  objectives  have  been  met  as  determined  by  the  TES  then  the 
students  may  move  on  to  written  posttests,  laboratory  experiences,  skill  testing  or 
the  next  unit  of  study.  If  certain  objectives  are  not  met  the  student  may  be  directed 
to  additional  study,  scheduled  for  a faculty  conference  or  recycled  to  the  learning 
resources  (in  a C MI  mode). 

(f)  The  computer  is  used  in  a C MI  mode  as  a monitoring  device  within 
the  TES  flow.  Computer  management  of  the  entire  teaching/learning  process  is 
not  part  of  the  model  in  Figure  1,  although  such  a dimension  is  certainly  possible. 

(3)  To  reiterate,  the  model  shown  above  emphasizes  the  independent  study 
concept  from  a "student  flow"  rather  than  from  a "systems  flow"  perspective. 

Within  this  model  the  instructional  designers  determine  where  and  in  what  dosages 
CAI,  CMI  or  DCI  can  best  be  utilized.  This  utilisation  is  determined  by  the  con- 
tent of  material  being  presented  and  the  instructional  strategy  best  suited  to  that 
content . 

d.  "Miniexperiments"  should  be  inserted  in  the  existing  project.  This  can  be 
accomplished  without  changing  (a)  the  time  schedule  for  the  project  or  (b)  the 
primary  thrust  of  the  evaluation. 

(1)  These  miniexperiments  provide  an  effective  mechanism  for  answering 
questions  which  are  crucial  for  implementation  of  future  training  systems  in  a 
cost  effective  fashion. 

(2)  For  example,  in  the  student  flow  model,  certain  task  elements  could  be 
presented  utilizing  non-computerized  learning  resources;  other  task  elements 
could  be  presented  using  CAI  as  a learning  resource.  In  one  version  a tutorial 
evaluation  system  on  the  computer  could  be  utilized  while  another  could  utilize  a 
written  evaluation  system.  Such  miniexperiments  might  help  cordon  off  those 
components  of  the  teaching/learning  process  or  student  flow  which  are  best  met 
by  CAI,  CMI  or  CDI.  Figure  2 offers  a conceptual  framework  for  several  mini- 
experiments aimed  at  the  most  effective  and  efficient  match  of  type  of  computer 
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involvement  with  components  of  the  teaching/learning  process  with  type  of  learning. 
Cost  accounting  and  manpower  studies  concerning  development  and  support  of  each 
of  the  options  is  crucial. 

(3)  Another  variation  for  a miniexperiment  might  be  to  allow  some  students 
to  enter  the  system  based  upon  pretests  placement  while  others  enter  based  upon 
their  own  self  assessment.  Such  a study  of  student  ability  to  accurately  self  assess 
needed  areas  of  study  could  prove  fruitful. 

(4)  Variations  of  the  CAI  support  team  structure  (see  pages  101,  103) 
would  provide  the  setting  for  additional  informative  miniexperiments. 

e.  A mechanism  must  be  devised  for  "shipping  a man  out"  at  the  point  his 
training  is  complete. 

(1)  If  the  incentive  of  early  completion  via  self-paced  individualized  study  is 
removed  or  negated  by  placement  of  the  student  in  a less  pleasant  duty  assignment 
until  the  next  class  rotation,  then  any  evaluation  of  project  effectiveness  will  be 
totally  muted.  Some  incentive  or  reward  system  for  early  completion  must  be 
instituted  if  any  cost  effectiveness  or  decrease  in  learning  time  is  to  be  realized 
in  project  ABACUS.  Given  an  accurate  CMI  system  and  well -developed  predictor 
equations  for  student  performance,  the  student  in  conjunction  with  his  advisor, 
must  be  able  to  plan  a projected  graduation  date  with  considerable  accuracy . ^ 

Early  graduation  must  be  rewarded  positively  (i.e. , given  advanced  training)  not 
negatively  (i.e. , given  additional  duty  assignments). 

(2)  Viewed  on  a broader  scope  the  Army  will  also  reap  rewards  from  pro- 
viding proper  exit  paths  from  training.  By  placing  the  student  immediately  on  the 
job  or  by  moving  him  quickly  to  his  next  training  area,  the  Army  saves  training  time 
and  potentially  training  dollars.  The  problems  in  the  current  exit  procedure  guar- 
antee no  possibility  of  time  savings. 

Recommendations  for  Design  and  Implementation  of  Future  Computerized  Training 
Systems 

To  this  writer,  two  areas  appear  as  prime  candidates  for  consideration  in  design 
of  computerized  Army  training  systems  for  the  future.  The  first  involves  the  per- 
sonnel support  for  production,  maintenance  and  updating  of  computerized  course- 
ware. Inherent  in  this  area  are  the  problems  of  career  path  and  incentive  for  mem- 
bers of  the  course  development  team.  The  second  involves  mechanisms  for 


5 

Technical  Documentation  for  the  Computer  Assisted  Instruction  Reporting 
System  (CAIRS),  The  Ohio  State  University  College  of  Medicine,  Division  of 
Computing  Services  (1973) 
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utilization  of  computerized  training  systems  in  the  continuing  education  of  Army 
personnel.  Both  recommendations  address  components  of  the  total  training  life 
cycle  which  appear  vulnerable  to  cost  reduction  with  a better  systems  arrangement. 

The  Course  Development  Team 

a.  The  Course  Development  Team  operating  currently  in  Project  ABACUS, 
(individuals  designing  and  developing  CAI,  CMI  or  CDI  materials)  is  composed  of: 

(1)  An  instructor  from  one  of  the  three  test  courses  (MOS  31E20,  Field  Radio 
Repair;  MOS  31J20,  Teletypewriter  Equipment  Repair;  and  MOS  35L20,  Avionics 
Communications  Systems  Repair)who  has  been  trained  in  the  CLASS  I authoring 
language,  the  instructional  model,  teaching  strategies  and  lesson  development 
techniques  and  procedures.  This  instructor  is  also  the  content  expert. 

(2)  A course  leader  coordinates  the  design  and  integration  of  CTS  in  the  over- 
all course  structure. 

(3)  In  some  situations  a secretary  enters  the  CLASS  I commands  at  the  com- 
puter terminal. 

(4)  Consultants  from  the  central  ABACUS  staff.  These  individuals  have 
expertise  in  instructional  strategy  design,  instructional  technology  and  CLASS  I 
coding. 

b.  Figure  3 offers  a diagram  of  the  interaction  of  the  members  of  the  current 
course  development  team. 


FIGURE  3 
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c.  Using  the  preliminary  instructional  model  for  CTS,  the  instructor  operates 
as  instructional  programmer  (I.  P. ) developing  new  CAI,  CMI  or  CDI  materials. 
Theoretically  the  I.  P.  is  to  follow  the  instructional  model  and  use  his  knowledge 


of  content  and  of  instructional  design  combined  with  input  from  the  Course  Leader 
and  Project  Consultant  to  determine  which  instructional  medium  to  use  for  each 
teaching/leaming  task.  The  expertise  required  in  this  process  is  probably  more 
than  the  I.  P.  is  prepared  to  exercise  alone  and  yet  in  current  project  implementa- 
tion the  I.  P.  is  given  such  responsibility.  This  consultant's  conclusion  is  that, 
by  default,  no  analysis  of  appropriate  "medium-message”  mix  is  being  performed 
and  that  the  Course  Development  Team  approach  in  the  current  project  must  be 
altered  to  afford  the  teaching/learning  perspective  delineated  on  page  103. 

d.  A second  significant  draw  back  exists  in  Figure  3 in  that  the  I.  P.  currently 
is  an  instructor  in  a course  area  who  has  been  retrained  as  described  above  to 
become  a developer  of  CAI,  CMI  and  CDI  materials.  As  such  he  must  learn  in- 
structional design  and  computer  programming.  Extrapolated  to  a broader  Army 
scale,  the  net  effect  of  this  manpower  retraining  is  staggering.  Not  only  are 
trained  teachers  being  unnecessarily  retrained  as  instructional  developers  and 
programmers,  but  each  new  I.  P.  begins  at  level  1 proficiency  in  instructional 
design  and  in  instructional  programming.  With  each  new  area  converted  to  the 
CTS  training  approach,  instructors  in  that  area  would  experience  the  growing 
and  learning  problems  faced  by  instructors  (instructional  programmers)  in  other 
areas — in  suort,  the  lesson  learned  at  each  implementation  site  would  not  be  fed 
forward  effectively. 

e.  An  alternative  course  development  team  structure  is  shown  in  Figure  4. 
This  model  depicts  the  major  actors  as  the  author  and  the  instructional  program- 
mer (I.  P. ).  Hence,  the  content  expertise  resides  in  the  author  (or  instructor) 
while  the  I.  P.  serves  as  the  primary  interface  between  the  author  and  the  other 
members  of  the  team.  The  author  remains  the  content  specialist,  the  teacher  in 
the  classroom  environment,  the  advisor  to  the  student  and  the  overseer  of  the 
computer  managed  instruction.  The  instructional  programmer: 

(1)  consults  with  the  author  regarding  the  objectives,  strategy  and  design 
of  materials, 

(2)  codes  and  inputs  the  material  in  the  CLASS  I language, 

(3)  works  with  the  system  programmer  to  design  new  course  facilities 
and/or  systems  functions  to  effect  a course  strategy, 

(4)  works  with  the  systems  analyst  to  design  student  progress  reports, 
course  item  analyses  as  required, 

(5)  arranges  for  students  to  review  new  CAI  courses, 

(6)  arranges  for  additional  content  reviewers  as  required. 
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Figure  4.  Proposed  course  development  team. 


(7)  consults  with  the  author  and  the  audiovisual  consultant  on  the  use  of 
audiovisuals  in  conjunction  with  the  CAI  courses, 

(8)  remains  the  on-going  caretaker,  reviewer,  revisor  and  troubleshooter 
for  the  course  once  it  is  made  available  for  student  usage  (this  includes  continual 
monitoring  of  student  response  reports)  and 

(9)  monitors  and  helps  interpret  statistics  and  student  reports  used  in  evalu- 
ation and  revision  of  course  materials . 

f.  Since  the  I.  P.  is  expected  to  be  a "jack-of-all-trades"  he  or  she  must  have 
available  "back-up"  resources.  Thus,  an  audiovisual  consultant,  educational 
strategy  consultant,  course  management  supervisor,  systems  programmer  and 
report  analyst  are  available  as  needed  to  assist  the  instructional  programmer 

. and/or  author  in  their  respective  areas , 

g.  The  I.  P.  assumes  responsibility  for  technical  knowledge  and  system  inter- 
face and  thus  frees  the  author -instructor  to  work  with  content  and  strategy.* 

Within  a course,  the  strategy,  the  lessons  and  the  points  for  the  computer  assist- 
ance are  defined  by  the  author  /instructor  in  consultation  with  the  I.  P. , the  course 
supervisor  and  the  educational  strategy  consultant.  The  instructional  programmer 
along  with  his  consulting  educational  strategist  and  author /instructor  becomes  an 
artist  who  helps  determine  which  components  of  the  teaching/learning  process 
should  involve  CAI,  CMI,  CDI  or  other  media.  The  I.  P.  helps  determine  when 
to  combine  testing  and  evaluation  (see  student  flow  page  97),  when  to  suggest  alter- 
native learning  resources,  and  how  to  efficiently  and  effectively  implement  the 
educational  strategy  for  a given  training  area  or  objective.** 

h.  The  educational  strategy  consultant  and  the  systems  support  staff  for  the 
instructional  programmer  (potentially  from  a central  Army  group  such  as  the 
current  ABACUS  central  staff)  could  be  called  upon  to  train  new  instructional  pro- 
grammers, to  assist  them  in  implementation  at  each  CTS  training  area  and  to  pro- 
vide continuing  consultation  as  needed.  Once  implementation  is  completed  some 
instructional  programmers  would  be  left  on  site  at  the  training  center  to  effect 



*This  concept  does  not  negate  the  need  for  more  refined  authoring  languages 
in  this  model.  Such  enhancements  would  only  make  the  I.  P.  's  more  productive. 

*, 

At  this  point  it  should  be  noted  that  the  division  of  responsibility  among  mem- 
bers of  the  development  team,  as  delineated  above  is  not  rigid.  In  fact,  care 
must  be  taken  to  avoid  overcompartmentalization  of  tasks,  since  the  team 
effort  is  crucial. 
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continuing  revisions,  troubleshooting  and  upgrade  of  the  CTS  system  materials. 
This  continuing  maintenance  would  be  done  in  concert  with  the  course  supervisor 
and  instructors  in  each  training  area.  As  a trained  technical  specialist  the  in- 
structional programmer  along  with  the  support  staff  shown  in  the  above  model 
could  move  rapidly  to  spearhead  implementation  of  the  CTS  concept  in  each  Army 
training  area. 

i.  The  above  model  has  been  used  for  some  time  at  The  Ohio  State  University 
College  of  Medicine.  Currently,  instructional  programmers  operating  in  the 
environment  of  the  model  produce  approximately  three  to  five  new  hours  of  CAI 
instructional  materials  per  month.  This  is  in  addition  to  responsibilities  for 
revision,  maintenance,  and  troubleshooting  for  some  200  operational  CAI  units. 
Figure  5 offers  average  course  development  times  by  type  of  CAI  and  assumes 
an  inexperienced  author  and  an  experienced  instructional  programmer.  The 
"types  of  CAI"  are  defined  below: 

(1)  Tutorial  Evaluation:  Simple  linear  programming  of  questions 

with  appropriate  feedback  to  individual  responses;  hints  or  coaching  sequences 
may  be  included. 

(2)  Tutorial  Evaluation  with  Options:  Simple  programming  of  instructional 
questions  which  permit  the  learner  a choice  of  sections,  questions  or  informa- 
tional material. 

(3)  Tutorial  Evaluation  with  Question  Branching:  Programming  of  questions 
and  sub -questions  which  further  individualize  the  material  according  to  prerequi- 
sites, objective  achievement  and  non -achievement;  assumes  standard  student  data 
base  retained. 

(4)  Diagnostic  Instruction:  Extensive  individualized  instructional  program- 
ming within  questions  or  groups  of  questions  which  permits  the  learner  and/or 
the  system  to  identify  learning  problems  and  have  the  opportunity  for  drill  and 
practice  or  prerequisite  "make-up";  assumes  an  expanded  student  data  base  with 
tagged  questions  and  branching  accordingly;  branching  increases  the  learner's 
control  of  the  activity. 

(5)  Individualized  Tracking:  Programming,  according  to  response  data 
which  allows  question  branching  with  appropriate  feedback  specific  to  a learner's 
background  and  prerequisite  and  objective  achievement;  the  tracking  concept  may 
be  utilized  to  allow  a learner  to  progress  through  the  material  according  tc  the 
response  and  thus  approach  a "quasi"  simulation  learning  activity. 

j.  As  can  be  seen,  the  author  time  involved  for  development  of  various  types 
of  CAI  utilization  would  be  significantly  greater  if  instructional  programming 
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interface  were  not  provided.  The  extent  of  the  time  savings  is  underestimated  in 
Figure  5,  since  instructional  programmer  time  is  based  upon  tha  past  expertise 
and  cumulative  experience  of  a pool  of  I.  P.  's. 


k.  The  significance  of  the  I.  P.  for  review  maintenance  and  revision  of  existing 
courseware  must  not  be  overlooked.  Not  only  is  this  individual  invaluable  in 
making  minor  changes,  adjustments  and  refinements  in  content,  but  this  person 
will  automatically  perform  all  necessary  system  upgrades,  conversions,  and  alter- 
ations in  the  courseware  which  should  be  transparent  to  the  user  and  the  ai^hor. 

As  an  example  of  the  time  consumed  in  one  such  maintenance  function,  Figure  6 
shows  the  average  time  spent  bj  an  instructional  programmer  and  management 
supervisor  or  author  in  conversion  of  course  units  from  a 100  character  lines  to 
70  character  lines.  The  figures  are  even  more  impressive  when  one  notes  that 
all  courses  first  passed  through  a master  convertor  which  performed  initial 
translations.  I.  P.  time  was  spent  in  refinement  and  in  realignment  of  changes 
which  could  not  be  properly  managed  by  the  program.  The  management  super- 
visor (author  review)  time  was  spent  for  changes  in  which  the  line  length  affected 
the  educational  presentation  and  validity  of  materials . Given  a)  the  total  man 
hours  required  for  upgrade  of  one  hour  of  interactive  CAI  (25.8  hours)  and  b) 
two  hundred  and  eighteen  course  units  averaging  two  hours  each,  there  were  four 
hundred  and  twenty  six  interactive  course  hours  of  OSU  materials  to  upgrade. 

Thus  11.24P  man  hours  or  281  man  weeks,  or  over  4 man  years  were  spent  in 
the  conversion.  Extrapolation  of  this  to  all  Army  training  areas  gives  some 
appreciation  for  the  amount  of  maintenance  and  revision  time  required  for  com- 
puterized teaching  materials. 

l.  And  finally,  three  very  important  side  effects  of  the  instructional  programmer 
and  support  team  concept  as  described  above  must  be  considered: 

(1)  Given  this  model,  educational  strategy  and  application  of  CAI  materials 
always  holds  first  priority  over  fancy  hardware  and  computer  systems . This  is 
due  to  the  fact  that  the  computer  support  individuals  (systems  programmers  and 
report  analyst)  exist  to  support  the  instructional  programmer  and  author  rather 
than  to  direct  them.  The  model  operates  to  meet  the  educational  needs  rather  than 
to  direct  them. 

(2)  Definition  of  what  is  to  be  develoDed  comes  from  outside  the  computer 
support  team  and  eliminates  the  tendency  of  the  computer  support  group  to  breed 
a life  of  its  own. 

(3)  Each  new  Army  training  area  or  course  development  effort  utilizes  the 
staff  and  expertise  in  the  central  support  unit.  Small  support  "fiefdoms”  each 
"recreating  the  wheel"  of  CAI,  CDI  or  CMI  are  not  allowed  to  form.  Each  effort 
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Figure  6.  Character  line  conversion. 
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builds  upon  the  knowledge  and  expertise  of  past  efforts;  lessons  learned  by  support 
staff  or  authors  on  one  project  or  in  one  training  arena  can  be  applied  immediately 
to  others . 

m.  A word  of  caution  must  be  rendered  In  any  model  which  suggests  separation 
of  computer  hardware  and  software  support  staff  from  the  instructional  course 
development  staff.  In  such  an  environment,  artificial  "walls  can  hinder  the  appro- 
priate communications  of  system  needs,  alterations  and  evolve  merit  Care  must 

be  taken  to  insure  that  although  the  instructional  programmer  and  nan -technical 
course  development  staff  function  m one  arena  and  the  compiaer  systems  design, 
software  and  hardware  support  individuals  function  in  another  arena,  the  two  are 
constantly  aware  of  concerns,  needs  and  restraints  of  the  other.  Clearly,  training 
needs  ought  to  dictate  the  computer  hardware  and  software  support  needs,  bia  at 
the  same  time  the  course  development  team  must  be  aware  of  the  constraints 
under  which  the  computer  system  support  group  functions. 

n.  The  above  model  does  not  necessarily  advocate  one  centralized  hardware 
and  software  computer  support  system.  The  general  model  is  one  of  separation, 
of  powers  of  computer  support  Individuals  from  instructional  design  support 
individuals,  regardless  of  whether  there  is  one  national  computer  network, 
regional  networks,  or  individual  compiler  systems  at  each  Army  training  center. 

o.  A potential  drawback  in  Army  use  of  the  above  model  is  that  no  long  term 
incentive.  Military  Occupational  Specialty  (M06)  or  career  path  exists  currently 
for  members  of  the  course  development  team.  Civilians  could  be  used  where  the 
criterion  of  ''world  vide  military  use"  does  not  dictate  a military  career  area. 

The  civilian  option  would  allow  for  higher  level  entry  grades  and  could  be  particu- 
larly applicable  to  the  educational  strategy  consultant  for  example. 

p.  An  appropriate  career  pattern  and  reward  track  for  instructional  program- 
mers might  include  advancement  to  warrant  officer  where  potentially  the  position 
of  chief  warrant  officer  for  support  of  training  systems  could  evolve  at  each  Army 
training  center.  This  warrant  officer  could  work  as  a team  leader  in  the  course 
development  process  and  could  supervise  implementation  and  operation  of  the  CTS 
support  system  at  the  training  center.  Alternatively  an  M06  of  instructional  pro- 
grammer might  be  justifiable  if  an  Army -wide  implementation  of  CTS  (pages  no-112) 
became  a reality.  A senior  instructional  programmer  (e.g. , Senior  Sergeant 

could  a)  supervise  groups  of  I.  P.  's  b)  offer  consultation  regarding  'If"  and  "how" 
to  incorporate  various  instructional  technologies  into  Army  training  and  c)  help 
integrate  CAI,  CMI,  and  CDI  into  the  other  instructional  technology  areas  (i.e., 
(AVTV)  in  the  Army). 

q.  For  the  Army  officer  involved  in  the  above  model,  a specialty  program 
similar  to  that  of  "Audiovisual  Instructional  Technology"  in  the  Officer  Personnel 
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Management  System  might  be  appropriate.  Optionally,  for  these  individuals  who 
have  existing  professional  expertise  in  instructional  strategy  and  modeling,  might 
not  a career  path  similar  to  that  of  the  Army  M.  D,  be  a feasible  solution? 

r.  In  summary,  the  problems  of  appropriate  mix  of  Army  and  civilian  personnel 
and  stabilization  and  career  path  for  Army  personnel  must  be  addressed. 

Extrapolation  of  the  CTS  Concept 

a.  The  second  long-term  recommendation  observed  as  relevant  by  this  con- 
sultant relates  to  extrapolation  of  the  CTS  concept  beyond  the  point  of  implementa- 
tion at  each  Army  training  center.  The  paragraphs  which  follow  are  admittedly 
long-term  in  prospectus;  still  the  potential  which  they  address  should  not  be  over- 
looked. First,  assume  as  given,  the  implementation  of  computerized  training 
systems  to  allow  for  truly  individualized  time-independent  movement  through  all 
major  Army  training  programs.  Furthermore,  assume  that  the  compiter  net- 
work supporting  the  training  system  is  national  n scope.  Under  these  conditions 
each  Army  installation  could  become  a training  center  for  all  content  areas  in  the 
CTS  national  system.  Army  personnel  could  complete  necessary  training  for  a 
given  area  without  physically  changing  installations.  Major  training  centers  would 
exist  only  where  learning  required  equipment,  tutors  or  other  resources  not  nor- 
mally found  or  cost-effectively  placed  on  any  Army  installation.  Some  courses 
(i.e. , Army  procedures  and  forms  design)  could  be  centrally  designed  and  admin- 
istered but  actually  taught  via  CTS  at  each  installation.  The  audiovisuals,  work- 
books, etc.  could  be  made  available  at  each  installation  just  as  tutors  and  student 
advisors  could  be  made  available  to  answer  questions  and  to  verify  student  pro- 
ficiency. Such  advisors  could  hold  regular  jobs  on  the  installation  and  consult  or 
tutors  only  part  time.  Other  courses  (i.e.,  helicopter  repair)  would  most  likely 
be  regionally  based  since  equipment  and  competent  instructors  would  not  be  avail- 
able at  every  Army  installation. 

b.  Technically,  however,  the  system  with  CMI  reporting,  computer  assisted 
instruction,  computer  directed  instruction  and  transportable  learning  components 
(i.e.,  tapes,  audiovisuals)  could  be  utilized  in  a learning  lab  environment  at  each 
training  installation.  Students  could  then  be  sent  to  the  nearest  installation  to  pro- 
ceed with  training  in  a specialty  area  (covering  as  many  content  areas  as  possible 
at  one  given  location).  Such  a decentralized  training  system  would  address  two 
very  important  problems  in  the  current  system. 

(1)  The  cost  and  personal  trauma  of  transfer  of  individuals  from  location 
to  location. 

(2)  The  "red  tape"  timing  problems  in  movement  of  individuals  from  one 
installation  to  another  upon  completion  of  a segment  of  training. 
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c.  The  author  must  note  that  only  the  most  careful  of  studies  will  determine 
whether  the  system  proposed  above  will  truly  address  the  problems  in  a cost 
effective  manner. 

d.  The  situation  proposed  above  is  not  totally  different  from  ideas  from  extra- 
polation of  the  basic  sciences  portion  of  M.  D.  training  to  remote  areas  where  no 
medical  school  exists.  In  the  WAMI  Project*’  basic  sciences  study  is  being  offered 
in  a four  state  area  (Washington,  Alaska,  Montana  and  Idaho)  from  one  central 
medical  school  base  at  the  University  of  Washington.  Knowledgeable  medical  course 
advisors  and  tutors  are  available  at  learning  centers  in  each  of  the  states  while  the 
instructional  materials  are  developed,  designed,  maintained  and  distributed  from 
the  central  site  at  the  University  of  Washington.  Upon  completion  of  basic  science 
training,  the  students  move  to  the  University  of  Washington  for  the  clinical  experi- 
ence where  work  with  patients  is  required. 

e.  Potentially  such  a centralized  computerized  training  system  could  piggy-back 
on  an  existing  national  network.7,  8 Experience  has  shown  that  the  system  load 
for  transfer  of  educational  materials  is  minor  in  a large  data  transmission  network 
(i.e.,  5 to  7 characters  per  second).  This  determination  would,  however,  have 

to  be  based  upon  accurate  figures  involving  the  number  of  students  to  be  trained, 
the  type  of  CAI  and  the  amount  of  materials  and  training  offered  via  CAI  and  other 
modes. 

f.  Potentially,  the  most  exciting  outcome  of  the  above  extrapolation  of  the 
computerized  training  system  concept  would  be  that  of  providing  an  on-going  data 
base  for  continuing  education.  By  having  materials  continually  available  to  every 
base  on  the  network,  the  operational  individual  (i.e.,  the  radio  repairman)  need 

0 

A proposal  to  the  Veterans  Administration  for  experimentation  in  Bio-Medical 
Communications  using  the  ATS-F  communications  satellite  and  ten  Veterans 
Administration  hospitals  in  the  Appalachian  region  submitted  jointly  by:  The 
Appalachian  Regional  Commission  and  the  Foundation  for  Applied  Communications 
Technology. 

7Lewis,  Jinnet  Fowles  and  Wooster,  Harold,  "Description  and  Utilization  of  the 
Lister  Hill  Center  CAI  Experimental  Network"  (unpublished  paper)  presented  at 
the  Association  for  the  Development  of  Computer  Based  Instructional  Systems, 
August,  1973. 


8Combs,  Bill,  "TYMNET:  A Distributed  Network",  Datamation,  July,  1973 


simply  come  to  the  CTS  system  periodically  for  testing,  recertification  or  re- 
study. Centralized  files  on  performance  and  proficiency  could  be  easily  main- 
tained for  consideration  in  promotions,  etc.  Furthermore,  increased  student 
awareness  of  his  own  continuing  education  needs  might  be  worthy  of  exploration. 
Individuals  could  be  given  the  opportunity  to  review  materials  at  any  time  and/or 
required  periodically  to  go  through  the  recertification  process  without  having  to 
leave  their  operational  installation  site. 

g.  Similar  experience  in  the  medical  environment  has  been  achieved  through 
the  efforts  of  the  Computer  Assisted  Instruction  Regional  Education  Network 
within  the  state  of  Ohio.9  This  network  of  community  hospitals  utilizes  materials 
developed  for  primary  training  of  nurses,  physicians,  and  allied  health  personnel. 
Continuing  education  credit  for  the  CAI  exercises  has  been  rendered  by  the  Ameri- 
can Medical  Association,  The  American  Dietetics  Association  and  other  such  or- 
ganizations. Thus,  the  physician  or  other  health  professional  in  a small  town  in 
Ohio  can  draw  upon  the  expertise  and  materials  .from  major  university  for  self- 
learning and  study  to  fulfill  requirements  for  recertification.  Currently,  a similar 
concept  for  recertification  of  internists  is  being  explored  by  the  American  Board 
of  Internal  Medicine.  Recertification  will  be  based  upon  performance  in  simu- 
lated case  studies  available  to  the  internist  in  his  local  community  via  a computer 
terminal. 


Summary 

a.  In  summary,  the  project  thrust  and  evaluation  perspective  should  be 
altered  to  view  the  type  and  scope  of  computer  involvement  as  determined  by  the 
needs  and  characteristics  of  the  teaching/learning  process,  rather  than  by  a 
predetermined  plan  which  superimposes  computing  indiscriminately  on  the 
teaching/learning  process  (pages  95,  96  ) and  student  exit  from  the  teaching/ 
learning  process  (page  100)  must  be  addressed. 

b.  Implementation  of  the  CTS  concept  beyond  Project  ABACUS  further  dictates 
review  and  restructuring  of  the  course  development  team  with  particular  attention 
given  to  career  path,  incentive  and  stability  for  Army  personnel  involved.  Extra- 
polation of  the  CTS  concept  to  Army-wide  training  and  continuing  education  affords 
potentially  the  most  exciting  cost  effective  and  productive  use  of  computer  based 
learning  in  the  Army  environment. 


9Pengov,  Ruann,  "The  Computer  as  an  Additional  Aid  in  Medical  Education",  Pro- 
ceedings of  the  Tenth  Annual  Council  Meetings  and  Conference  of  EDUCOM.  1974. 

19Harless,  William  G.,  Drennon,  GaryG.,  Marxer,  JohnJ.,  Root,  Judith  A., 
Wilson,  Linda  and  Miller,  George,  "CASE — A Natural  Language  Computer  Model" 
Computing  in  Biological  Medicine,  Vol  3,  pp.  227-246,  printed  in  Great  Britain. 
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Background  and  Statement  of  the  Problem 

Technological  developments  within  the  computer  industry  are  advancing  at  a 
rapid  pace,  particularly  in  the  hardware  area.  Even  in  a period  of  general  econo- 
mic recession  and  inflation,  these  developments  have  been  (and  can  be  expected  to 
continue  to  be)  significant  enough  to  expand  existing  markets  and  open  new  market 
areas  with  a downward  trend  in  prices.  This  brief  paper  will  highlight  some  of 
these  developments  which  should  have  an  impact  on  Army  training  systems  of  the 
future  and  make  some  general  recommendations  on  areas  which  should  be  considered 
in  future  system  designs . 

Discussion 

a.  The  most  significant  development  in  the  electronics /computer  industries  of 
this  decade  has  undoubtedly  been  the  microprocessor,  or  CPU  on  a chip.  It  can  be 
safely  said  that  this  development  will  revolutionize  the  electronics  industry  and 
have  a major  impact  on  the  computer  industry.  Just  as  the  introduction  of  mini- 
computers expanded  the  computer  industry  and  market  by  an  order  of  magnitude, 
the  microprocessor  can  be  expected  to  have  a similar  order  of  magnitude  impact 
in  the  near  future  and  several  orders  of  magnitude  additional  impact  in  the  long 
term.  It  is  probably  also  safe  to  say  that  whereas  computer  technology  and  appli- 
cations have  been  applied  by  only  a small  portion  of  e!3Ctrical/electronic  engineers 
in  the  past,  they  will  be  common  and  essential  tools  of  the  vast  majority  of  these 
engineers  in  the  future.  Few  areas  now  employing  electromechanical  or  electronic 
control  systems  of  any  degree  of  complexity  will  not  fall  into  the  domain  of  micro- 
processors in  the  years  ahead 

b.  The  vast  markets  that  can  thus  be  envisioned  for  microprocessors  give  rise 
to  anticipated  low  costs  for  these  devices.  Already  it  is  possible  to  buy  complete 
microcomputer  systems  (including  CPU  chip,  minimal  memory  and  I/O  logic)  for 
as  little  as  $29.95  in  quantity  (INTEL  4040).  The  limiting  cost  factors  are  gen- 
erally pins,  interconnectors,  packaging,  etc.,  rather  than  the  devices  themselves . 
With  chip  component /device  densities  doubling  almost  every  two  years,  we  can  look 
for  continued  improvements  in  these  areas . 
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c .  Another  related  technology  development  area  is  solid  state  (semiconductor) 
memories.  Here  too  benefits  are  being  realized  from  the  increasing  density  trend 
in  integrated  circuits . The  day  is  not  too  far  away  when  semiconductor  memories 
will  edge  out  magnetic  cores  for  main  memories  on  an  economic  basis  and  efforts 
now  underway  on  solid  state  BORAMS  (Block  Oriented  Random  Access  Memories) 
foresee  them  replacing  many  disc  storage  systems  as  well.  Within  the  time  frame 

(considered  for  this  paper  it  is  possible  to  project  solid  state  (semiconductor)  mem- 
ories being  readily  available  up  to,  perhaps,  the  megabyte  level.  Truly  mass  mem- 
ories will  probably  continue  to  be  dominated  by  discs,  with  solid  state  systems  being 
employed  in  a limited  number  of  special  systems.  Some  speed  improvements  can 
also  be  foreseen. 

d.  Display  technology  can  be  expected  to  make  significant  advances.  In  partic- 
ular plasma,  LED  (Light  Emitting  Diodes),  and  thin  film  technologies  can  be  ex- 
pected to  yield  improved  performance  characteristics  (density,  color,  and  gray 
scale)  at  affordable  prices . Another  area  which  should  move  into  the  functionally 
useful  and  affordable  category  is  voice  input /output.  Discrete  (word  at  a time, 
limited  vocabulary)  speech  recognition  systems  are  in  use  today  in  limited  appli- 
cation areas  and  work  is  continuing  at  a high  level  in  continuous  speech  recognition. 
Voice  synthesis  units  are  also  in  use  today  as  output  devices  and  are  continually 
improving. 

e.  Given  these  technological  trends  what  are  the  implications  for  the  Army  and 
its  future  design  of  training  systems  ? 

(1)  Overall  system  architectures  will  probably  shift  somewhat  toward  the 
decentralized/distributed  concept  and  away  from  the  large  centralized  system.  Con- 
current reductions  in  communications  costs  will  continue  to  make  centralized  sys- 
tems viable,  particularly  where  resource  sharing  with  nontraining  systems  is  de- 
sirable. Nevertheless,  microprocessor  (microcomputer)  developments  and  lower 
cost  memories  will  have  a significant  impact  on  terminal  design  and  cost . Concepts 
now  employed  in  the  ABACUS  distributed  miniapproach  can  be  further  exploited 
through  "more  intelligent"  terminals  reducing  requirements  now  placed  on  the  dis- 
play controllers  and  the  system  controller.  Portions  of  the  data  base  system  can  be 
considered  for  decentralization  as  well.  It  should  be  possible  to  proceed  in  the 
direction  of  modular  systems  with  varying  numbers  of  terminals;  the  system  could 
readily  accommodate  an  expanded  number  of  terminals  since  the  new  terminals 
would  bring  with  them  most  of  the  additional  processing  power  required.  At  the 
low  end,  it  should  be  possible  to  achieve  virtually  stand-alone  systems  with  as  few 
as  one  or  two  terminals  employing  tape  cassettes  for  course  loading  and  administra- 
tive files.  This  would  be  paiticularly  attractice  for  remote  training  requirements, 
3uch  as  Army  Reserves  and  National  Guard,  permitting  them  to  utilize  the  same 
courseware  developed  for  active  Army  use. 
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(2)  Terminal  design  will  be  impacted  in  several  ways.  Display  technology 
will  probably  facilitate  improved  graphical  capabilities  and  student  interaction. 

These  should  be  planned  for  in  terms  of  course  development.  The  particular  tech- 
nology that  proves  to  be  most  suitable  in  the  long  run  is  not  a major  consideration  at 
this  time  since  their  capabilties  are  similar.  It  should  be  noted  that  PM,  ART  ADS 
currently  has  an  advanced  development  project  under  way  for  the  design  and  fabrica- 
tion of  an  Interactive  Computer  Presentation  Panel  (ICPP),  intended  to  provide  an 
interactive  display  device  to  superimpose  graphics  on  standard  Army  maps.  The 
technology  being  developed  (hardware  and  software)  will  have  potential  applications 
in  training,  and  the  IC  PP  itself  may  be  of  use  in  team  training  situations . 

(3)  Microprocessors  and  inexpensive  semiconductor  memory  can  provide 
more  intelligent  terminal  capabilities  as  previously  discussed.  Further,  micro- 
processors can  readily  be  employed  as  multi -media  device  controllers  to  enhance 
terminal  performance. 

(4)  Associated  with  CAI/CMI  terminals,  special  purpose  devices  configured 
as  instrumented  (on-line)  mockups  of  selected  military  equipment  can  be  fabricated 
employing  microprocessors  to  assist  in  limited  equipment  simulation.  Such  devices 
would  be  applicable  both  to  individual  and  team  training  situations . The  capability 

to  tie  such  devices  into  the  overall  system  in  an  on-line  mode  would  significantly 
enhance  equipment  oriented  training  functions,  further  reducing  requirements  for 
training  personnel  and  improving  individual  problem  diagnosis  and  subsequent  re- 
medial actions. 

(5)  Audio  (speech)  input/output  has  utility  in  some  areas,  but  should  be 
used  sparingly.  Audio  input  from  the  student  may  be  of  use  where  an  effort  is  being 
made  to  completely  simulate  an  operational  environment.  This  is  another  area 
where  an  ongoing  effort  by  PM,  ART  ADS  may  be  of  assistance.  A current  advanced 
development  project  will  result  in  the  delivery  to  the  Army  of  a Discrete  Speech 
Word  Recognition  System  (WRS)  aimed  at  allowing  voice  input  to  tactical  data  sys- 
tems (e.g.,fire  mission  requests  to  the  Tactical  Fire  Direction  System).  This 
system  may  be  available  for  limited  experimental  use  in  training  if  an  appropriate 
test  situation  were  devised  on  a noninterfering  basis  with  ARTADS . 

(6)  Future  Army  training  systems  should  make  greater  use  of  CAI  while 
retaining  CMI  functions.  Frojected  terminal  economics  should  make  this  possible. 

For  the  CMI  functions,  however,  a hard  copy  capability  should  be  added  to  the  ter- 
minal. Detailed  individualized  instructions  prepared  for  a student's  off-line  usage 
could  then  be  delivered  in  a form  to  facilitate  his  compliance  with  those  instructions . 

(7)  Performance  of  the  system  in  a CMI  mode  can  be  further  enhanced  by 
the  utilization  of  source  data  automation  to  the  maximum  extent  possible.  This 

is  to  free  the  instructors/testors/evaluators  or  other  administrative  personnel 


from  having  to  enter  off-line  generated  data  into  the  system.  On-line  equipment 
simulaters  are  one  example  of  how  this  can  be  accomplished.  Other  special  purpose 
devices Aerminals  should  also  be  considered  where  appropriate.  Intermediate  docu- 
mentation should  be  minimal,  if  not  eliminated  entirely. 

(8)  Finally  (as  an  aside),  some  thoughts  on  the  application  of  educational 
technology  to  complex  equipment/system  oriented  training  are  offered.  One  area 
in  which  technology  can  not  be  expected  to  lead  to  significant  improvements  is  lead- 
time  requirements  for  the  development  of  new  hardware  and/or  software  systems. 
Hence  training  requirements  and  implications  must  be  considered  from  the  earliest 
possible  time  in  the  development  cycle  of  any  new  equipment  or  system.  This  will 
enable  early  decisions  as  to  the  optimum  method  of  accomplishing  required  training. 
In  some  systems  (those  having  significant  internal  computer  facilities,  for  instance) 
it  may  be  possible  to  utilize  the  system  itself  (or  perhaps  with  minimal  augmentation) 
for  training  purposes;  others  may  be  best  taught  via  system  hardware  simulators; 
still  others  may  require  only  good  software  packages . The  important  thing  is  that 
any  required  development  should  take  place  concurrently  with  the  target  equipment, 
consistent  with  the  risks  associated  with  potential  changes  in  that  equipment.  The 
design  of  any  Army  training  system  can  be  effective  only  if  it  adequately  considers 
the  nature  of  the  training  for  which  it  will  be  utilized  when  it  becomes  operational . 

Conclusions  and  Recommendations 

a.  In  summary,  future  technological  developments  can  be  expected  to  create 
a number  of  opportunities  for  significant  improvements  in  Army  training  systems . 
Microprocessors,  semiconductor  memories  and  display  technology  all  are  candi- 
dates for  exploitation  in  future  designs. 

b . Specific  recommendations  offered  for  consideration  are  the  following: 

(1)  Continue  to  pursue  distributed  system  architectures . Evolve  toward 
a modular  incremental  system  with  intelligent  terminals  and  minimum  overhead 
equipment  requirements. 

(2)  As  an  objective,  the  intelligent  terminal  high  performance  and  should 
lead  toward  a minimal  capability  stand-alone  system. 

(3)  Terminal  design  should  consider  the  use  of  high-performance  micro- 
processors and  semiconductor  memory  to  provide  local  processing  capabilities, 
improved  graphics  and  multimedia  control  functions . 

(4)  Microprocessor  driven  equipment  mockups  and  simulators  should  be 
considered  for  improved  performance  oriented  training  capabilities,  both  in  indi- 
vidual and  team  training  situations . 
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Introduction 


a.  The  following  discussion  of  the  proposed  ITSM  covers  four  basic  facets: 

(1)  a formal  management  mechanism  for  continued  feed-in  and  feedback  (cyber- 
netic) relationships  among  research,  development  and  operational  implementation 
of  innovations;  (2)  establishment  of  a National  Center  for  Instructional  Technology 
to  coordinate  and  accomplish  the  required  R&D  for  innovative  training  techniques; 

(3)  continued  development  of  new  hardware  technology  (especially  computers)  as 

a means  of  addressing  the  manifold  problems  of  training  and  upgrading  requisite 
training  materials;  and  (4)  conceptualizing  the  ITSM  as  an  evolving  system  with 
a template  that  could  aid  all  military  Services. 

b.  I will  first  deal  with  the  last  item  concerning  an  example  of  the  way  such 

a functional  organization  might  fit  into  a National  Center  within  the  ITSM.  Figure  1 
illustrates  the  functions  in  the  following  way.  TRADOC  would  submit,  through  its 
operational  review  board,  new  projects  which  would  be  approved  and  selected  fa- 
research  or  development  at  the  National  Center.  Personnel  would  be  experts  in  a 
potential  development  effort  and  would  be  sent  on  a TDY  status  to  the  National 
Center.  Here,  if  they  did  not  have  training  in  the  use  of  innovative  instructional 
technology,  these  developers  or  instructors  would  be  trained  ia  techniques  such 
as  programmed  instruction,  computer  based  learning  developments  and  whatever 
other  advanced  technologies  might  be  available  at  the  Center.  Administrators, 
as  shown  in  Figure  1,  might  also  be  sent  to  the  Center  for  training  in  the  new  roles 
that  they  would  take  on  given  the  use  of  the  innovation  which  would  be  under  develop- 
ment (such  as  counseling  modes  available  to  them  in  the  use  of  CAI  or  computer- 
based  simulations,  etc.).  At  the  Center  proper,  there  would  be  a staff  nucleus 
maintained  which  would  be  headed  by  a mcnager  to  oversee  the  various  R&D  pro- 
jects. The  development  side  of  the  Center  would  concentrate  on  exactly  what  is 
within  Figure  1,  namely,  development  of  a particular  curriculum  effort.  This 
would  be  accomplished  through  the  use  of  personnel  from  the  field,  the  experts  in 
the  particular  subject  matter,  and  the  instructional  technology  experts  of  the  Center 
staff.  As  the  new  project  goes  through  the  development  cycle,  it  may  well  be  that 
potential  spin-offs  in  the  form  of  new  research  hypotheses  would  be  generated.  For 
example,  it  might  be  that  the  uses  of  graphics  come  up  as  a question  for  research 


^•"This  paper  was  prepared  by  the  author  as  an  independent  consulting  effort  and  in 
no  way  represents  the  opinion  or  policy  of  the  Human  Resources  Research  Organi- 
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to  answer  in  a particular  class  of  training.  This  hypothesis  then  would  be  dis- 
cussed with  the  research  staff  at  the  Center  and,  if  found  appropriate  a research 
project  would  be  generated  to  cover  strategies  that  might  be  generated  to  cover 
strategies  that  might  oe  employed  to  deal  with  the  question  of  graphics.  Secondly, 
should  the  requirement  be  for  new  technology  to  display  such  graphics  as  are 
indicated  in  the  hypothesis , then  the  hardware  technology  staff  would  be  consulted 
and  the  project  might  design  hardware  as  well  as  instructional  strategy  research 
as  part  of  its  coordinated  effort. 

c.  Following  the  project  development  through  to  completion,  once  the  new 
effort  is  completed  within  the  Center — that  is,  the  subject  matter  has  been  thoroughly 
gone  through  in  terms  of  behavioral  objectives,  specifications  of  media  require- 
ments, clearly  laid  out  relationships  between  prerequisite  structure  components 

of  the  subject  matter — it  then  would  become  the  province  of  the  quality  control 
staff  at  the  Center.  They  would  put  the  materials  together  in  the  proper  packaging 
and  through  the  dissemination  capability  at  the  Center  would  send  the  packaged 
innovation  (the  example  shows  computer  based  learning  materials  (CBLM)  simply 
because  this  is  the  most  advanced  innovation  currently  being  worked  on)  to  the 
various  user  sites.  At  the  user  sites,  whether  they  be  Army  Schools  or  Units — 
and  if  we  look  to  the  future  there  will  be  a greater  concentration  on  Unit  instruc- 
tion— the  second  component  of  the  dissemination  staff,  namely  the  staff  at  the 
user  site,  would  help  the  particular  Unit  or  School  prepare  the  site  for  efficient 
and  effective  utilization  of  the  new  subject  matter.  This  would  involve  technical 
support  of  the  local  staff,  as  well  as  training  them  in  the  educational  or  training 
implementation  of  the  new  package  to  be  incorporated  into  the  instruction  avail- 
able at  the  user  site.  This  would  also  involve  demonstration  and  sampling  of 
the  way  in  which  the  innovation  works.  As  Figure  1 indicates,  delivery  of  the 
material,  should  it  be  computer  based,  might  well  consider  the  network  possi- 
bility from  a central  source  to  telephone  lines  or  microwave  or  other  such  com- 
munication links  as  might  be  deemed  appropriate  to  the  potential  operational  users. 
Moreover,  at  the  site  .jself,  once  the  dissemination  staff  working  closely  with  the 
unit  site  gets  familiarity  with  the  particular  innovation,  new  requirements  might 
be  generated.  It  is  important  to  note  that  the  new  requirements  are  not  neces- 
sarily going  to  be  simply  a function  of  the  experience  with  a particular  innovation, 
but  may  well  come  from  experience  on  the  job  or  at  the  school.  In  any  event, 
the  dissemination  staff  at  the  user  site  would  help  in  packaging  the  new  require- 
ments and  putting  them  into  a form  which  would  be  understandable  in  common 
language  to  the  Center  staff. 

d.  One  more  item  not  covered  yet  would  be  the  function  of  pilot  testing  and 
revision.  As  indicated  in  the  dotted  lines  covering  the  relationship  between  the 
quality  control  staff  and  operational  sites,  the  pilot  testing  would  take  place  once 
the  packaging  has  been  accomplished  by  the  quality  control  staff  at  the  potential 


122 


January  6,  1975 


SUBJECT:  Consultant  Report  to  CTS 

TO:  Dr.  Joseph  S.  Ward,  Educational  Technology 

Organizations  and  Systems  Research  Lab 
U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
1300  Wilson  Blvd. 

Arlington,  Virginia  22309 


FROM: 


Dr.  Robert  J.  Seidel 


The  problems  addressed  are  potential  short-  and  long-term  goals  in  terms 
of  the  two  sets  of  considerations — the  original  intent  and  design  of  CTS  as  com- 
pared to  the  revised  intent  and  design.  The  original  concept  of  the  CTS  was  that 
a prototype  delivery  system  would  be  for  CAI  tutorial  instruction  and  that  the 
location  would  be  at  Ft.  Monmouth.New  Jersey  Signal  School.  With  the  change 
in  the  location  of  the  syster..  now  to  be  implemented  at  Ft.  Gordon,  there  have 
come  a host  of  other  changes  which  necessitate  comments  before  making  recom- 
mendations. First  of  all,  there  have  been  a significant  number  of  changes  in  the 
intention  to  use  the  CTS  for  CMI  purposes  as  opposed  to  CAI  purposes.  The 
staffing  has  been  changed  considerably,  particularly  at  the  management  level 
at  Ft.  Gordon. 

The  questions  that  arise  then  relate  to  whether  or  not  the  previously  designed 
delivery  system  will  be  appropriate  for  a different  set  of  purposes  and,  secondly, 
whether  or  not  the  original  schedule  can  be  maintained  for  final  delivery  of  the 
CTS  prototype.  The  original  version  of  the  system  was  to  have  a fair  degree  of 
graphics  and  a tutorial  interactive  kind  of  environment  in  the  main  for  instructional 
purposes.  However,  from  our  review  of  the  current  status  of  the  project,  a solely 
interactive  graphics  CRT  capability  may  not  be  as  appropriate  for  the  CMI  environ- 
ment which  is  now  foreseen.  Another  question  to  be  reconsidered  concerns  the 
possible  provision  of  an  automated  scheduler  for  allocation  of  all  instructional 
resources  (other  media,  reference  manuals,  etc.).  Additionally,  a CMI  oriented 
approach  involves  considering  the  addition  of  a second  kind  of  terminal,  different 
staffing  requirements  for  development,  etc.  The  larger  question,  also  relevant 
from  the  beginning,  of  transfer  to  other  schools  and  other  courses,  must  now  be 
reexamined  in  the  light  of  the  change  in  implementation  to  take  place  at  Ft.  Gordon. 

In  like  manner,  from  a configurational  iewpoint:  Is  the  current  design  going 
to  be  upward  compatible  with  a viable,  new  system  alternative  for  follow-on 
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development  and  implementation  (e.g.,  the  hardware  configuration,  per  se,  local 
versus  teleprocessed,  maxi -mini -combinations,  system  software,  other  off-the- 
self  considerations)?  For  example  as  noted  above,  the  terminals  have  already 
been  decided  upon  through  the  RFP  as  principally  alphanumeric  with  some  graphic 
capability.  However,  much  of  the  Army  technical  and  combat  training  which  exist 
now  and  probably  more  so  in  the  future  lend  themselves  to  much  more  use  of 
graphics  than  considered  at  the  current  time  (especially  for  simulation  work  with 
expensive  equipment  configurations).  This  question  also  relates  to  the  selection 
of  course  materials  under  the  current  Project  ABACUS,  as  it  relates  to  general 
utility  in  the  Army  and  in  other  Services,  as  well.  For  example,  if  the  configura- 
tion changed  others  would  have  to  rewrite  the  course  materials  unless  it  is  clearly 
shown  that  there  is  a reasonable  and  feasible  relationship  to  some  follow-on  system. 
I have  considered  some  of  these  hardware  and  software  decision-making  questions 
in  two  earlier  publications  and  suggest  they  might  be  consulted  <or  additional  in- 
formation. (See  Seidel,  1973,  and  1971,  pp.  10-12,  attached.) 

With  these  provisos  in  mind,  I will  now  address  my  comments  to  what  seem 
to  be  relevant  yellow  flags  raised  at  the  two-day  consultant  conferened.  I will 
try  to  confine  myself  to  items  which  seem  relevant  to  the  Systems  Group,  however, 
many  of  these  may  seem  may  seem  to  overlap  with  the  Training  Group's  interest 
and  probably  there  will  be  a good  deal  of  overlap  with  other  consultant  comments, 
as  well. 

Short-Term  Goals 


The  first  point  to  make  here  is  that  it  is  fully  recognized  that  GTE  Sylvania 
is  providing  the  basic  hardware  and  systems  software  as  per  the  RFP  specifi- 
cations. I assume  that  this  is  not  going  to  change.  Questions  can  be  raised, 
however,  about  the  relationship  between  the  course  materials  that  will  be  written 
to  test  out  the  system's  software  (e.g.,  exercising  the  graphics  capability,  data 
base  access)  and  the  overall  evaluation  plan  which  is  relevant  to  evaluating  the 
cost/effectiveness  of  Project  ABACUS.  To  date,  there  does  not  seem  to  be  an 
established  plan  for  relating  the  application  software  which  is  internal  to  the  course 
and  student  evaluation  to  the  acceptance  test  of  the  external  software,  or  system 
software . It  may,  therefore,  be  quite  relevant  to  consider  revision  of  the  -Evalu- 
ation Plan  to  include  incorporation  of  a link-up  between  the  application  software  and 
the  system  software.  This  seems  to  become  especially  critical  when  we  are  con- 
sidering a system  which  is  now  CMI-oriented  as  opposed  to  CAl-oriented.  Even 
if  it  is  not  desirable  to  revise  in  any  radical  way  the  Class  1 language  at  this 
point,  it  may  be  extremely  important  to  note  the  potential  requirements  for  follow- 
on  systems  and  where  particular  difficulties  may  arise  (e.g.,  author  formatting, 
methods  for  data  access)  given  the  current  command  structure  of  the  language. 
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Another  extremely  critical  item  is  proper  course  selection.  The  fact  is  that 
much  of  the  research  and  development  in  CAI  in  the  past  has  shown  quite  clearly 
that  you  can  save  20%  to  40%  of  a student's  time  in  a course  by  (a)  systems  en- 
gineering of  the  course,  and  (b)  using  some  form  of  programmed  instruction.  It 
is  important  to  note  that  the  candidate  courses  chosen  at  Ft.  Gordon  are  already 
all  self-paced.  They  therefore  have  shown,  as  indicated  by  our  hosts,  a good  deal 
of  savings,  somewhere  on  the  order  of  20%  already.  In  addition,  the  major  portion 
of  the  current  selected  courses  are  hands-on,  off-line,  some  65%  for  example 
being  performance -oriented  portions  of  the  course.  Moreover,  only  5%  of  the 
instruction  is  planned  to  be  interactive  through  the  use  of  the  terminal.  From 
our  introduction  to  the  three  courses,  dropout  rates  are  extremely  low  already. 

It  is  therefore  extremely  questionable  as  to  whether  or  not  the  addition  of  the 
computer  can  yield  a greater  savings  in  student  weeks  per  course  to  warrant  its 
inclusion  as  an  add-on  dollar  investment  in  the  delivery  of  the  course  to  the  student. 

For  political  purposes,  I would  suggest  that  serious  consideration  be  given  to 
choosing  non-self-paced  courses  as  the  initial  candidates.  At  least  one,  high- 
density  course  with  a relatively  heavy  dropout  rate  would  help.  I doubt  that  this 
change  will  cause  much  of  a delay  in  the  project's  attainment  of  its  final  goal, 
since  the  materials  currently  being  written  are  being  prepared  without  knowledge 
and  without  full  implementation  of  the  software  and  hardware  of  the  CTS.  Note 
also  that  the  acceptance  tests  have  been  delayed  a number  of  months  because  of 
some  already  existing  deficiencies  in  the  delivery  from  GTE-Sylvania.  Indeed, 
it  may  well  be  unrealistic  to  attempt  to  retain  the  original  delivery  date  for  the 
completion  of  Project  ABACUS. 

Along  the  same  lines,  to  accomplish  dramatic  dollar  savings,  I would  also 
suggest  that  the  newly  selected  courses  include  a much  larger  percentage  of  soft- 
skilled  kinds  of  instruction.  This  is  particularly  appropriate  because  the  system 
does  not  seem  to  lend  itself  too  well  currently  for  troubleshooting  or  other  equip- 
ment simulations.  (Of  course,  exploitation  of  the  graphics  authoring  capability 
may  overcome  this  lack  to  some  degree . ) 

Reconsideration  should  be  given  also  to  the  question  of  whether  in  the  128  - 
terminal  system,  all  terminals  have  to  be  interactive  terminals  as  opposed  to 
splitting  them  between  some  interactive  CRTs  and  some  batch  type  printer  ter- 
minal which  would  be  less  expensive  and  probably  just  as  appropriate.  The 
Air  Force  Advanced  Instructional  System  development  has  a mix  of  such  ter- 
minals and  perhaps  can  be  sought  for  guidance  in  this  design  area.  The  Air 
Force  should  be  able  to  provide  some  additional  guidance  in  the  area  of  file 
structures  and  other  resource  allocations  in  a CMI  environment  since  this  is 
the  specific  thrust  of  their  project. 
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Related  to  the  cost-effectiveness  discussion  above  is  the  fact  that  the  one  other 
way  to  make  dollar  savings  in  easy  measurable  terms  is  through  the  elimination 
of  waste  time  for  graduates  in  individualized  curricula.  The  whole  question  of 
early  assignment  and  prediction  of  early  graduation  would  seem  to  require  much 
more  detailed  inclusion  in  the  evaluation  plan.  Public  Law  51  can  be  circumvented. 
HumRRO  papers  a couple  of  years  ago  addressed  that  very  problem.  Secondly,  a 
paper  by  Wagner  of  HumRRO  dealing  with  the  problems  of  modeling  early  predic- 
tion also  suggested  techniques  then  that  today  might  be  applied  in  the  CTS  project 
in  order  to  better  the  predictive  capability  for  individualized  graduation  and  there- 
fore individualized  assignment . IL  would  be  significant  then,  if  this  project  with 
the  high  visibility  that  it  already  attracts  can  get  the  DCSPER  responsible  indi- 
vidual (e.g.,  BG  Stevenson),  to  cooperate  in  a mutually  beneficial  development 
at  the  national  level  to  utilize  more  effectively  the  existing  computerized  assign- 
ment system.  At  least  in  the  near  term  efforts  should  be  made  to  get  the  CTS 
graduates  from  Ft.  Gordon  assigned  to  their  job  on  a higher  priority  basis  than 
a student  from  any  other  school,  thuj  eliminating  all  existing  delays.  Dollar 
savings  could  easily  be  shown. 

Long-Term  Recommendations 

In  this  section  I will  divide  my  remarks  into  three  categories  dealing  with: 
potential  future  orientation  of  training  programs  in  the  Army,  future  delivery 
systems  for  new  training  orientation,  and  thirdly  the  relationship  between  those 
two  new  developments  and  a redirected  curriculum  research  and  development  cap- 
ability within  the  Army . 

The  first,  rhetoric  dealing  with  the  new  look  of  the  volunteer  Army,  stresses 
the  small  number  of  personnel,  the  heterogeniety  of  personnel  and  the  requirement 
for  instantaneous  upgrade  or  change  in  readiness  posture.  What  follows  is  an 
attempt  to  deal  with  this  set  of  requirements  in  a realistic  manner.  A concept  of 
Army  training  presented  by  Colonel  Howard  at  the  Project  ABACUS  Conference 
concerned  a three-dimensional  matrix  dealing  with  the  focus  on  units,  teams  or 
individuals;  locations  at  schools,  units  or  National  Guard  and  Reserve;  and  with 
various  kinds  of  curricula.  The  old  premise  under  which  training  was  conducted 
seemed  to  be  to  bring  all  the  trainees  to  the  schools  first,  and  then  giving  them 
OJT  to  make  up  the  difference  between  what  the  school  could  not  provide  and  what 
was  required  on  the  job. 

I would  suggest  that  we  address  Colonel  Howard's  matrix  for  the  future  by 
trying  a reversal  of  that  premise  and  ask  what  this  implies  for  the  nature  of  Army 
training.  Obviously,  it  has  the  potential  to  bring  the  school  to  the  student  instead 
of  the  student  to  the  school  and  make  training  more  on-the-job  than  in -school 
(therefore  increasing  unit  and  individual  training).  Industry  (e.g. , IBM)  has 
already  cost -justified  CAI  training  of  their  personnel  in  this  manner.  I would 
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emphasize  the  value  of  three  attributes:  flexibility,  decentralization,  minimal 
travel  for  the  Army.  One  of  the  hoped  for  advantages  certainly  would  be  lowered 
costs  of  training.  An  integral  element  in  such  an  approach  would  be  heavy  use  of 
the  computer  to  facilitate  delivery  of  instruction,  to  lower  course  development 
costs,  and  to  aid  personnel  assignment. 

Clearly,  the  implication  is  for  (1)  minimal  travel  and  per  diem  costs  by  students 
(2)  less  numbers  of  schools,  consolidated  schools,  fewer  instructors,  and  there- 
fore lower  cosls,  (3)  more  spare  equipment  would  be  required  for  OJT,  which 
probably  would  increase  costs  somewhat  but  (4)  the  latter  probably  would  be  offset 
by  higher  motivation  for  the  students,  since  the  training  and  the  job  would  be  prin- 
cipally collocated  and  therefore  more  meaningful  training  to  the  trainee . This 
would  also  imply  that  feed -in  from  training  to  job  for  the  student,  and  feedback 
from  the  job  to  revision  of  training  programs  would  be  much  easier,  therefore 
facilitating  course  update  and  better  upgrading  of  job  requirements  for  the  solider . 

It  also  would  mean  that  refresher  training  and  secondary  MOS  training  could  be 
provided,  given  that  computer  configurations  with  versatile  terminals  are  available. 
For  initial  training  outside  CONUS,  PL-51  will  have  to  be  considered  but  this 
should  not  be  insurmountable . 

Realistically,  what  kind  of  a formal  mechanism  would  make  such  a proposal 
possible?  For  one  thing,  an  amalgamation  of  the  management  of  development 
projects  involving  computer  aids  and  other  technology  aids  to  training  at  one  site 
would  facilitate  the  coordinated  monitoring  of  a nationwide  and  worldwide  US  Army 
effort.  A beginning  apparently  is  already  under  informal  consideration  (e.g. , 
placing  CATTS,  CTS  and  other  technologies  under  a single  directorate  at  Ft.  Eustis). 
I would  suggest  that  what  is  further  needed  is  a formal  management  mechanism  by 
which  new  R&D  projects  are  intiaied  from  the  field;  research  and  curriculum 
development  projects  can  be  conducted  and  completed  at  a center,  avoiding  thereby 
duplication  of  efforts  with  limited  resources;  and  the  available  materials  or  find- 
ings given  back  to  the  operational  field  units  and  schools  where  applicable  for 
pilot  testing.  Once  field  tested,  materials  could  be  delivered  with  computer 
assistance  insuring  a minimal  amount  of  delay  from  concept  to  finished  product. 

I think  that  this  can  be  done  with  a functional  orientation  to  efforts  conducted  at  a 
centralized  training  research  and  development  center  (National  Center  for  Instruc- 
tional Technology).  To  accomplish  these  functions  I propose  an  Integrated  Train- 
ing System  Model  (ITSM)  which  is  described  in  detail  in  the  attached  paper. 


user  sites.  Thus,  the  final  package  disseminated  by  the  Center  staff  would  not 
take  place  until  there  had  been  pilot  testing  and  revision  cycles  had  been  gone 
through  to  insure  a criterion  having  been  met,  such  as  the  90-90  or  whatever 
else  it  might  be. 

e.  The  framework  envisioned  for  the  structure  then  of  this  Center  would  enable 
the  Army  to  follow  the  guidelines  embodied  in  the  systems  development  cycle  con- 
cept, and  wc.'ld  further  link  comprehensive  R&D  programs  with  operational  users. 
Parenthetically,  it  might  be  added  that  resources  may  become  even  more  limited 
than  is  currently  the  case.  Therefore  requirements  would  exist  for  greater  attempts 
at  eliminating  duplication  across  the  Services.  Consequently,  a DoD  level 
approach  to  this  functional  orientation  would  be  desirable,  ft  might  well  include 
ARPA  as  the  coordinating  body  and  would  operate  under  the  same  logic  except  on 

a larger  scale.  The  importance  here  is  that  with  this  approach  the  Army  could 
certainly  provide  the  leadership  in  this  direction. 

f . The  overview  presented  above  will  be  discussed  in  greater  detail  in  the 
section  concerning  the  system  description  of  ITSM  over  the  long  haul.  Specific 
management  operations  and  research  and  development  functions  will  be  covered. 

Objectives 

a.  To  provide  a framework  and  capability  for  (a)  making  the  transition  to 
cost /effective  unit -oriented  training,  and  (b)  continued  improvement  and  wide-- 

spread  implementation  of  individualized  training  facilities  and  systems. 

i 

l 

b.  To  optimize  within  the  ITSM  framework  the  allocation  of  resources  through 
sharing  of  Research,  Development  and  Operational  facilities  and  personnel. 

c.  To  enable  architectural  growth  of  a CAI  computer  system  which  allows 
for  a wide  range  of  capabilities  to  be  implemented,  depending  on  the  requirements 
of  individual  users  (schools  or  units),  and  on  requirements  of  new  courses  of 
instruction. 

d.  To  use  the  results  of  early  Army  operational  experience  with  the  system, 
to  continually  improve  the  system  design  and  implementation. 

e.  To  provide  the  capability,  if  desired,  to  capitalize  upon  ARPA  network 
facilities . 
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(a)  The  basic  management  concept  integrates  Operations  and  RfcD  under  a third 
party  Management  group.  The  system  is  shown  in  simplest  form  in  Figure  2.  For  conveni- 
ence, the  system  will  be  called  the  Integrated  Training  System  Model  or  ITSM. 


Figure  2.  Basic  management  concept. 


(b)  A tentative  organizational  description  of  the  system  is  shown  on  the 
following  page  as  Figure  3. 
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Figure  3.  Pilot  organization  and  major  functions 
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(2)  This  model  illustrates  the  capabilities  required  to  accomplish  the  specific 
activities  described  under  Pilot  Implementation. 

(3)  For  purposes  of  short-term  goals,  the  R&D  Support  function  can  be  pro- 
visionally viewed  as  shown.  The  Management  function  may  be  vested  in  a mission- 
oriented  committee.  This  committee  might  have  the  following  composition: 

. ARI  Representative 

. Management  Information  Systems  Expert 
. Subject-Matter  Expert  from  the  Pilot  Site 
. Personnel  Expert  from  the  Pilot  Site 
. DCSIT  TRADOC  Representative 

(4)  Operations  functions  are  performed  by  the  participating  instructional  staff 
of  the  Pilot  Site.  These  suggestions  are  intended  as  context  for  interpreting  the 
diagram,  and  not  as  firm  recommendations.  For  discussion  purposes  these  func- 
tions can  be  assumed  to  be  carried  out  at  Ft.  Gordon  with  the  implementation  under 
Project  ABACUS.  Such  an  approach  provides  a framework  for- efficient  solutions 

to  implementing  innovations:  physically,  financially  and  operationally. 

b.  Technical  Concept 

(1)  The  technical  approach  is  characterized  primarilyby  flexibility — both 
in  the  design  of  CAI  computer  systems  and  in  the  development  process  for  such 
systems.  For  example,  a new,  multi-mini  system  might  be  designed  as  a true 
multiprocessing  system.  The  system  would  distribute  its  resources  dynamically 
among  the  several  miniprocessors  so  that  individual  tasks  can  be  performed  in 
the  most  efficient  manner.  Thus,  these  processors  would  be  designed  to  share 
functions;  and  a variable  number  of  processors  may  be  used,  depending  on  com- 
plexities and  number  of  terminals  to  meet  school  and  course  requirements  Find- 
ings of  Project  ABACUS  would  be  fed  back  from  the  pilot  implementation  site  to 
the  National  Center  in  order  that  they  be  incorporated  into  this  new  design  effort. 

(2)  To  illustrate  further,  the  early  operational  capability  feature  of  ITSM  • 
would  enable  Army  personnel  to  develop  skills  and  materials  on  terminals  that 
might  be  used  in  a tested  system.  HumRRO  experience  has  shown  that  CAI 
materials  cannot  be  developed  adequately  on  the  drawing  boards ; that  hand-on  ex- 
perimentation-with  operational  features  of  the  instruction  and  pedagogy,  on  a live 
system,  are  essential. 
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(3)  An  early  operational  capability  such  as  Project  ABACUS  and  its  follow- 
ons  in  the  proposed  model  would  enable  the  Army  to  make  in  vivo  comparisons  of 
alternative  technologies . How  might  an  application  of  this  approach  be  made  ? As 
an  example  of  terminal  comparisons  consider  that  the  Plasma  panel  is  a very  pro- 
mising new  technology.  The  Color  Half-Tone  Area  Graphics  Environment  (CHARGE) 
terminal  is  a device  (HumRRO)  based  on  a new  concept. 

(4)  The  types  of  data  available  at  present,  on  which  to  make  a comparative 
evaluation  of  promising  terminals,  varies  from  case  to  case  (e.g.,  CMI,  CAI,  high- 
graphics  use,  etc.).  The  ongoing  studies  are  providing  useful  data  rerPlasma  Panel 
and  PLATO  IV . Project  ABACUS  will  yield  data  from  a more  intensive  prototype 
application  at  Ft.  Gordon.  However,  in  comparing  costs,  not  only  initial  cost  but 
operational  costs  must  be  considered.  Operational  costs  include  reliability.  When 
the  eventual  number  of  Army  terminal  installations  is  considered  (say  8, 000  or 
even  twice  that),  operational  costs  and  reliability  may  make  a difference  of  mil- 
lions of  dollars  between  one  terminal  type  and  another,  even  when  initial  purchase 
costs  are  comparable. 

(5)  Performance  characteristics  among  terminals  vary.  The  problem  in 
comparing  performance  on  paper  is  the  inability  to  predict  what  the  actual  per- 
formance will  be  under  operational  conditions.  The  Army  is  interested  in  capi- 
talizing on  the  latest  technology  developments  to  achieve  an  inexpensive,  high  per- 
formance, reliable  terminal.  However,  the  terminals  using  this  latest  technology 
must  be  thoroughly  tested  under  operational  conditions  to  assure  that  the  perform- 
ance characteristics  will  remain  satisfactory  with  repeated  use  under  full  load 
conditions . 

(6)  Another  dimension  of  performance,  aside  from  engineering  reliability, 

is  the  knowledge  of  the  performance  capabilities  that  are  required  by  the  application. 

(7)  The  CHARGE  terminal,  for  example,  will  permit  dynamic  color  simu- 
lation of  combat  terrain  or  equipment  characteristics  and  allow  for  continuous 
interaction.  Different  levels  of  detail  and  varied  perspectives  will  occur  as  if  the 
trainee  were  actually  working  in  and  around  the  real  environment  (and  equipment). 
With  the  continually  increasing  complexities  of  new  weapons  systems  which  the 
soldier  in  the  field  will  have  to  operate  and  maintain,  the  real  cost -performance 
payoff  from  using  computers  in  training  will  come  from  developing  and  testing  new 
training  concepts  not  hevetofcr  e available  in  the  traditional  instructional  setting. 

(8)  Another  way  in  which  terminals  must  be  compared  is  the  environmental 
conditions  required  for  operation.  Ambient  light  and  noise,  air  temperature, 
floor  space,  physical  configurations  of  space— combined  effects  of  all  of  these 

on  the  effectiveness  of  the  learning  environment — can  only  be  learned  through 
operational  experience. 
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(9)  A final  area  for  comparison  is  that  of  computer  system  support  require- 
ments. The  way  in  which  a particular  terminal  system  receives  its  input,  generates 
characters,  etc. , affects  the  nature  and  amount  of  computer  capability  required  for 
optimal  operation  of  the  terminal.  This  computer  support  must  be  figured  into  the 
overall  cost  of  the  terminal  operation. 

(10)  With  the  ITSM  approach,  a base  of  comparative  data  could  be  built  up  of 
experience  with  various  terminals  operating  in  the  same  physical,  pedogogical  and 
technical  environment.  This  empirical  base  would  enable  the  Army  to  select  the 
optimal  terminal  not  only  for  the  pilot  system  but  for  later  installations  as  well. 

c.  Transition 


(1)  A short-term,  mission-oriented  ITSM  might  be  employed  to  carry  out  the 
functions  and  activities  described  under  Project  ABACUS.  Over  time,  the  ITSM 
organization  is  expected  to  stabilize,  take  on  expanded  missions,  and  develop  the 
in-house  capabilities  required  to  achieve  its  long-term  objectives . 
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(2)  Transition  activities  will  be  planned  and  implemented  by  the  Army.  These 
activities  should  represent  a logical  extension  of  activities  initiated  under  Project 
ABACUS.  They  should  result  in  an  Army  in-house  capability  fully  responsive  to 
emerging  and  future  requirements  for  all  training  innovations.  The  computer  appli- 
cations are  chosen  here  because  of  their  immediate  value  and  because  they  represent 
the  leading  edge  of  individualized  training  innovations. 

< 

(3)  Activities  illustrating  the  intended  nature  of  the  transition  are  given  below. 

(a)  Designate  a DA  Steering  or  Project  Advisory  Group  to  study  and 
recommend  the  nature  of  the  permanent  ITSM  structure.  This  group  would  have 
membership  representing  the  interests  of  DCSPER,  OCRD,  DMIS,  and  TRADOC. 

It  could  be  an  added  function  of  the  existing  Project  ABACUS  SAG. 

(b)  Establish  missions  and  functions  for  the  permanent  organization. 

(c)  Assign  transition  personnel  to  develop  interim  TD  or  TOE,  and 
to  identify  permanent  staff. 

(4)  Depending  upon  study  result,  the  permanent  organization  may  take  a 
variety  of  forms . 

(a)  A central  physical  entity,  such  as  a National  Center  for  Instructional 
Technology,  including  facilities  for  instructor  training,  and  a communications  net- 
work (e.g. , field  representatives)  encompassing  selected  TRADOC  schools.  (This 
alternative  is  illustrated  in  Figure  4.) 


135 


(b)  A regional  system,  analogous  to  the  "proponent  school"  concept, 
organized  by  computer  applications,  with  both  an  inter -regional  and  satellite- 
school  communications  network. 

(c)  A decentralized  system  organized  by  subject-matter  and/or  MOS 
applications . 

(3)  Transition  personnel  would  discharge  their  missions  within  the  context 
of  Project  ABACUS.  Recommendations  for  a permanent  structure,  its  mission 
and  functions,  will  perforce  be  influenced  by  operational  experience  with  the  pilot 
system. 

(6)  From  the  emerging  experience,  it  may  be  possible  to  explore  new  options 
for  operational  and  R&D  components.  For  example,  operational  experience  may 
justify,  in  some  cases,  selection  of  one  or  more  maxicomputers,  centrally  located 
with  minicomputers  at  the  training  sites.  Where  appropriate,  the  central  compu- 
ter^) may  store  course  logic  or  text,  cr  provide  computing  capability,  and  inter- 
face with  the  rainiconfiguration  through  a system  comparable  to  the  current  ARPA 
IMP,  or  Interface  Message  Processor.  This  approach  is  intended  to  yield  greater 
performance  at  equal  cost.1 

d.  System  Description -Long-Term  Evaluation 

(1)  It  is  premature  to  project  the  exact  structural  nature  of  the  emerging 
ITSM.  However,  the  pilot  implementation  plan  described  earlier  suggested  major 
functions  that  are  expected  to  remain  important  for  an  Army-wide  ITSM.  As  be- 
fore, these  relate  to  management,  operations,  and  support  (R&D).  The  long-term 
success  of  the  ITSM  will  depend  heavily  upon  the  integrity  and  cohesiveness  with 
which  these  three  functions  are  maintained.  In  order  to  insure  that  this  takes 
place,  it  is  proposed  that  overall  authority  for  management  and  coordination  be 
vested  in  a small  committee  with  a rotating  chairmanship.  This  group  would  be 
composed  of  DA  level  representatives  from  DCSPER,  TRADOC,  OCRD  and  DMIS. 
This  i3  illustrated  in  Figure  4. 

(2)  This  section  suggests  some  functions  to  be  performed  by  the  three  com- 
ponents in  the  long-term  ITSM.  Figure  4 illsutrates  how  these  functions  would 

be  implemented  in  terms  of  both  the  flow  of  products  through  a National  Center, 
as  well  as  personnel.  Figure  4 should  be  used  as  a companion  to  Figure  1.  The 


"Current  Status  of  Computer-Administered  Instruction  Work  Under  Project  IMPACT, 
by  Robert  J.  Seidel,  Human  Resources  Research  Organization,  Alexandria,  VA., 
Professional  Paper  18-72,  July  1972. 
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Management  & Coordination 
DCSPER  OCRD 

OMIS  TRADOC 


Illustrative  flow  of  products  and  personnel  through  the  ITSM. 


added  information  in  Figure  4 is  thai  it  distinguishes  between  the  manpower  flow 
through  the  Army  Training  System  and  out  to  its  f;eld  units,  from  the  flow  of  pro- 
ducts to  be  used  by  these  people  in  the  field.  Thus,  the  bottom  portion  of  the  figure 
shows  the  influx  of  manpower,  perhaps  conceived  of  as  the  new  volunteer  Army, 
into  various  schools  and  units  for  appropriate  training.  The  manpower  would  then 
be  allocated  to  take  on  either  operational  courses  or  some  pilot  courses,  as  the 
case  may  be.  They  graduate  from  these  courses  and  are  assigned  through  the  pipe- 
line to  various  jobs . Requirements,  vis-a-vis  assignments,  would  clearly  come 
under  the  province  of  DCSPER  and  they  could  facilitate,  through  their  individualized 
assignment  capability,  the  allocation  of  new  manpower  into  various  units  and  schools. 
The  development  and  research  aspects  of  the  Center  are  somewhat  telescoped  in 
Figure  4,  but  they  nevertheless  contain  the  same  functions  in  the  form  of  a product 
flow  as  dees  the  illustration  in  Figure  1. 

(3)  The  remaining  items  then  to  be  covered  concern  specific  examples  of 
functions  to  be  covered  under  Research,  Development,  Management  and  Operational 
Use  of  the  Personnel  and  Products  within  the  proposed  ITSM.  These  follow. 

e.  Management 

(1)  Provide  overall  direction  and  establish  Army  standards. 

(2)  Select  pilot  schools;  coordinate  selection  criteria. 

(3)  Select  pilot  courses;  coordinate  selection  criteria. 

(4)  Insure  adequate  funding. 

(5)  Set  milestones  for  course  development,  computer  and  terminal  delivery 
and  installation,  and  software  development. 

(6)  Coordinate  priorities. 

(7)  Manage  the  packaging  and  dissemination  of  completed  computer- 
oriented  curricular  materials . 

(8)  Sponsor  information  exchange  conferences. 

(9)  Interface  with  other  agencies  within  the  Army  and  across  services . 

(10)  Provide  career  incentives  for  Curriculum  Developers  and  Instructors. 
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f.  Operations 


(1)  Provide  operational  test  sites  for  prototype  products  (hardware,  soft- 
ware, instructional  materials  and  strategies,  and  operating  procedures). 


(2)  Implement  and  maintain  prototype  systems. 


(3)  Provide  evaluative  information  at  various  levels,  e.g.,  cost/effective 
data  on  operating  systems  and  terminals,  and  other  feedback  regarding  course 
materials,  student  management,  and  assignment  procedures. 


(4)  Maintain  liaison  with  support  and  management  elements  through  such 
mechanisms  as  clearinghouses  and  centralized  system  support  capabilities. 

g.  Research 

(1)  Conduct  experimental  studies  in: 

(a)  alternate  instructional  management  strategies; 

(b)  techniques  for  the  systematic  development  and  revisions  of  instruc- 
tional materials;  and 

(c)  alternate  hardware  and  software  designs . 

(2)  Participate  inthe  design,  test,  and  evaluation  of  alternate  computer  con- 
figurations. 

(3)  Develop  alternate  models  for  the  management  of  individualized  training 
in  the  Army. 

h.  Development 

(1)  Develop,  test,  revise,  and  disseminate  computer -oriented  curricular 
materials  of  all  appropriate  types  (e.g.,  mainline  tutorials,  problem-solving, 
gaming,  drill  and  practice,  and  simulations). 

(2)  Provide  quality  control  standards  for  the  development,  documentation, 
and  dissemination  of  such  materials. 

(3)  Specify  the  parameters  for  an  individualized  training  management  system 
(e.g.,  student  scheduling,  staff  incentives,  accounting  procedures,  and  indivi- 
dualized graduation  and  assignment). 
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(4)  Specify  parameters  for  implementable  delivery  systems  (e.g.,  type  of 
hardware,  number  of  terminal/school,  and  software  facilities). 

(5)  Conduct  continuing  formative  evaluation  studies  in  school  environments. 

(6)  Provide  instructor  training  in  the  use  of  computer-oriented  curricular 
materials . 

(7)  Results  of  the  recent  survey  (by  Rich  and  Van  Pelt,  1974)1  of  Army  CAI 
reveal  fragmented  efforts  with  no  apparent  coordination,  very  similar  to  the  cottage 
industry  in  civilian  education.2  The  ITSM  approach  including  the  proposed  National 
Center  for  Instructional  Technology  would  provide  the  necessary  management  and 
technological  directions  for  the  Army  to  maintain  leadership  among  the  Services 

in  training  R&D.  It  would  also  yield  a feasible  model  for  making  the  transition 
from  a totally  school -based  operational  training  system  to  an  advanced  cost-effec- 
tive, unit -oriented  program. 

i.  ITSM  Potential  for  DoD 

(1)  Finally,  as  indicated  earlier,  the  same  logic  which  makes  the  ITSM 
appealing  within  the  Army  could  be  applied  at  the  DoD  level  on  a larger  scale. 

This  section  describes  a possible  extension  of  tne  ITSM  to  include  a Joint  Services 
Human  Resources  Institute. 

(2)  The  basic  purpose  of  such  an  institute  would  be  to  coordinate  and  develop 
a cohesive  research  and  development  program  relevant  to  the  efficient  use  and 
increased  capabilities  for  the  individual  military  person.  Problems  which  have 
beset  behavioral  science  efforts  tcldate  within  the  military  services  have  in  large 
part  stemmed  from  the  need  to  compete  with  one  another  for  limited  funds  and  to 
attempt  duplicative  or  parallel  developments  when  at  the  outset  the  funds  were  not 
sufficient  to  accomplish  even  partial  goals  which  were  set  by  the  particular  Ser- 
vices. What  this  means  is  that  each  Service  will  have  an  opportunity  to  contribute 
ideas,  concepts,  directions  relevant  to  its  perceived  unique  needs,  while  at  the 
same  time  benefitting  from  a unified,  multidisciplinary  effort  physically  located 

in  one  structure  and  providing  the  flexibility,  growth  potential  and  continuity  in 
resources.  The  perceived  necessity  for  studying  all  facets  of  a training  problem 


1 "Survey  of  Computer  Applications  in  Army  Training,  " by  Joseph  J.  Rich  and 
Kermit  B.  Van  Pelt,  Computerized  Training  Systems  Project,  Ft.  Monmouth,  NJ, 
Interim  Report,  August  1974,  CTS-TR-74-3. 

2Learning  Alternatives  in  U.  S.  Education:  Where  Student  and  Computer  Meet, 
by  Hunter,  Kastner,  Rubin  and  Seidel,  Educational  Technology  Publications, 

Inc.,  January  1975. 
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by  each  Service  will  be  eliminated.  The  hardware  or  software  subsystems  which 
would  be  necessary  to  deal  with  any  use  of  the  computer  or  other  educational  tech- 
nological innovations  can  be  handled  just  once  without  the  necessity  for  reinventing 
different  concepts — the  same  concepts — at  different  locations  and  in  different  Ser- 
vices . Evaluation  of  emerging  techniques  and  technologies  can  be  made  for  the 
benefit  of  all  in  a coherent  manner.  Thus,  for  example,  the  Plasma  panel  which 
might  be  considered  the  most  innovative  hardware  feature  in  the  way  of  student 
terminals  for  computer -administered  instruction  could  be  compared  dispassionately 
in  a deliberate  series  of  studies  with  other  potential  competitive  innovations  such 
as  the  storage  tube  approach  to  improvement  of  CRTs  and  graphic  capabilities. 
Exploitation  of  the  ARPANET  as  a delivery  system  for  the  Services  would  be  a 
natural  task  in  this  context. 


(3)  The  framework  envisioned  for  the  structure  of  this  institute  would  follow 
the  guidelines  embodied  in  the  systems  development  cycle  concept  and  would  indeed, 
furthermore,  link  comprehensive  research  and  development  programs  with  regional 
applications  research  centers  representing  the  four  Services.  This  means  that  the 
initial  steps  in  the  cycle,  that  is  the  determination  of  the  needs  of  the  various  Ser- 
vices, would  be  provided  by  them  as  feed-in  to  the  Central  Research  and  Develop- 
ment Institute  (CRDI).  Following  this,  representatives  of  the  Services  involved 
with  the  particular  problem  would  translate,  at  the  Central  Institute,  the  needs 
into  firm  objectives  for  exploratory  investigation.  Alternative  experimental  con- 
figurations would  be  tried  out  and  finally,  when  the  step  of  implementation  is 
reached  in  the  systems  development  effort,  a preliminary  development  would  be 
tested  within  the  Regional  Applications  Center  of  the  original  requesting  Service. 
During  the  time  of  development,  assessment  would  be  continually  made  concerning 
the  applicability  of  a particular  problem  and  potential  solution  for  the  other  ser- 
vices. And  when  the  point  of  implementation  is  reached,  the  continued  guidance 
during  the  systems  development  cycle  provided  by  the  other  interested  Services 
could  also  lead  to  a simultaneous  implementation  at  another  Regional  Center  rele- 
vant to  the  other  interested  Service(s).  A good  example  of  this  type  of  joint  involve- 
ment with  mutually  beneficial  results  is  the  problem  of  dealing  with  individualized 
graduation  and  assignment  to  a particular  job  or  MOS  following  training.  The  over- 
all model,  a type  of  traffic  control  model,  most  probably  would  be  the  same  for  all 
the  Services.  What  would  differ  would  be  the  data  points,  the  MOS  category,  or 
the  particular  parameters  for  assigning  individuals  to  a given  military  occupational 
speciality  versus  some  other  possible  category  of  job  relevance.  In  addition,  cer- 
tain characteristics  of  schools  would  be  unique  from  Service  to  Service.  However, 
in  the  main  the  overall  nature  of  individualized  flow  through  an  instructional  system 
(the  algorithm)  with  individualized  assignment  following  would  pretty  much  be 
equivalent  in  form  regardless  of  Service.  Clearly,  the  joining  of  forces  would  per- 
mit a much  more  massive  attack  on  the  problem  and  also  the  Services  would  benefit 
from  the  input  that  the  others  could  provide  as  well  as  themselves.  (Both  in  terms 
of  personnel  with  given  expertise,  as  well  as  facilities  and  other  resource  cap- 
abilities. ) 
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(4)  The  proposed  structure  would  be  as  follows . The  Central  Research 
and  Development  Institute  would  be  comprised  of  a cadre  of  representatives, 
technical  and  administrative,  from  each  of  the  Services.  For  the  most  part  these 
would  be  civilians  to  provide  the  necessary  continuity  of  effort.  At  the  Central 
Institute  there  wocld  also  be  capability  for  visiung  civilian  researchers.  Manage- 
ment of  the  Center  would  be  flexible  and  permit  the  letting  of  contracts  to  conduct 
many  of  the  studies  identified  by  the  Services  as  necessary  to  provide  for  their 
unique  problems.  The  management  of  the  Regional  Centers  would  be  mission- 
oriented  and  provided  by  the  particular  Service  for  whom  that  Center  is  created. 

The  researchable  ideas  and  problem  should  flow  in  two  directions.  Thus,  Regional 
Centers  as  a result  of  their  more  closely  allied  operational  contacts  would  be  able 
to  provide  insights  for  problem  identification,  the  solution  of  which  might  best  be 
satisfied  at  the  CRDI.  Moreover,  it  is  important  that  a bi-directional  feedback 
system  be  provided  for  in  this  network  arrangement.  Obviously,  the  importance 
of  this  overall  concept  is  to  provide  the  necessary  linkages  between  research, 
development  and  operational  utilization.  While  the  emphasis  obviously  is  to  be 
placed  upon  efficient  personnel  utilization  and  career  development,  the  areas  for 
investigation  must  include  related  fields  bearing  directly  upon  these  problem  areas. 
Thus,  for  example,  as  noted  above  in  the  field  of  CAI  language  development,  sys- 
tems software,  computer  developments  and  student  terminal  improvements  and 
modifications  must  be  considered. 

(5)  A number  of  considerations  remain  to  be  answered.  Some  of  the  more 
important  ones  are  who  will  direct  the  policies  and  planning  of  this  network  ? Will 
it  be  direct  from  OASD  Manpower,  or  will  there  be  a type  of  Joint  Chiefs  of  Staff 
arrangement  ? Will  the  annual  funding  for  such  a network  be  an  add-on  item  to 
each  Service's  R&D  budget,  or  will  it  be  composed  of  contributions  provided  from 
the  already  existing  annual  budgets  ? What  will  be  the  management  relationships 
between  the  Regional  Centers  and  the  CRDI?  What  will  be  the  nature  of  the  relation- 
ships between  in-house  work  and  contracting  studies? 

(6)  Once  these  questions  are  answered,  a DoD-wide  ITSM  could  become  a 
feasible  reality.  In  an  analogous  way  to  the  Army  approach,  the  proposed  Joint 
Services  Human  Resources  Institute  would  provide  the  necessary  management  and 
technological  leadership  to  accomplish  a cost -beneficial  model  for  a continuing, 
innovative  DoD  level  training  system. 
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research  areas,  both  long  and  short 
term,  that  in  the  opinion  of  this  consultant  are  imperative  to  the  success  of  Project 
ABACUS.  The  areas  are  all  related  to  the  capabilities  and  facilities  of  the  computer 
or  its  peripheral  equipment.  All  of  these  items  have  been  found  to  be  essential 
in  the  preparation  of  smooth  running  and  meaningful  computer  supported  instruction. 
Research  relating  to  each  should  prove  fruitful . Each  heading  in  each  numbered 
paragraph  indicates  a research  area  to  be  investigated.  Rationale  for  the  selection 
cf  such. areas  follows: 

Hardware 

a.  Programming  Languages.  Since  dependence  should  not  be  placed  on  pro- 
grammers who  have  little  or  no  experience  with  either  the  subject  being  developed 
or  the  pedagogical  procedures  that  are  the  most  satisfactory,  instructors  should 
be  trained  to  do  their  own  programming.  This  calls  for  the  use  of  a simple  pro- 
gramming language,  yet  one  which  has  the  necessary  power  and  versatility.  EX- 
TENDED BASIC  has  been  found  to  be  eminently  satisfactory  for  non -graphic  pro- 
grams. Now,  for  full  graphic  capabilities,  APL  is  generally  recommended.  Con- 
tinuing research  into  program  language  improvements  is  a necessity. 

b . Response  Time.  Computers  tend  to  slow  down  as  the  student  load  in- 
creases. In  choosing  the  hardware,  consideration  must  be  given  to  the  heaviest 
anticipated  load  so  that  a very  fast  response  can  be  expected.  Boredom  sets  in 
very  quickly  when  students  have  to  wait  for  continued  program  execution  after 
each  input.  Reduction  of  response  times  under  heavy  computer  loadings  is  a 
requirement  for  large  scale  systems. 

c.  Reliability.  Down-time  must  be  reduced  to  a minimum.  For  most  effi- 
cient use  of  terminals,  student  time-scheduling  is  of  paramount  importance.  When 
a student  loses  a session  due  to  computer  malfunction,  even  temporary  standby 
conditions,  it  is  difficult  if  not  impossible  to  make  it  up. 

d.  Backup.  The  system  should  make  it  possible  for  instructors  or  depart- 
ments to  copy  completed  non-volatile  programs  on  magnetic  tape  (or  a suitable 
substitute)  to  avoid  loss  of  the  programs  if  files  are  destroyed  during  a crash. 

e.  Terminal-to-Computer  Linkage.  Although  the  use  of  telecouplers  and 
modems  is  a useful  expedient  when  the  system  is  not  hard-wired,  it  is  very  ex- 
pensive and  inconvenient.  Any  planned  installation  should  take  into  account  the 
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need  for  a suitable  cable  for  this  linkage.  This  is  a matter  of  judiciously  selecting 
the  locations  of  the  CPU  and  the  peripherals  with  respect  to  the  terminal  sites. 
Investigation  for  solutions  of  this  problem  is  a fundamental  of  all  planning  stages 
in  extensive  installations . 

f.  High-8  peed  Printing.  Although  many  CAI  procedures  depend  only  on  ter- 
minal printouts,  the  availability  of  a high-speed  line  printer  is  a necessity  if  only 
for  backup  purposes.  This  applies  particularly  to  drills  and  test;  regardless  of  the 
number  of  such  items  stored  on  disc,  every  instructor  must  have  available  pre- 
printed copies  of  the  software  in  the  event  of  system  failure  when  drills  and  tests 
are  due  to  be  administered. 

g.  student  Terminals.  The  structure  of  the  student  terminals  for  and  to 
optimize  training  requirements  is  another  arena.  Video  versus  hard  copy  is  the 
big  question.  At  present,  it  is  easy  for  a student  to  carry  out  copies  of  tests  and 
problem  exercises  if  a hard-copy  terminal  is  used.  Instructors  should  have  access, 
however,  to  either  a hard-copy  terminal  or  a video  terminal  equipped  with  a termal 
printer  or  substitute,  ft  must  be  made  very  easy  for  instructors  to  obtain  student, 
class,  and  course  reports  from  the  computer's  files  and  equally  easy  for  students 
working  with  video  terminals  to  obtain  hard  copy  when  required. 

h.  Editing  Facility.  This  is  a matter  of  hardware  selection  as  well  as  pro- 
gramming. A highly  competent  intrinsic  editing  system  should  be  part  of  the  inven- 
tory of  the  facilities  of  the  system.  (For  example,  the  XEROX  SIGMA  6 and  the 
XEROX  SIGMA  9 are  both  equipped  with  a remarkable  editing  facility).  Changes 

in  course  content,  an  inevitable  outcome  of  data  accumulation,  will  be  postponed 
or  never  made  at  all  unless  the  editing  facility  is  very  fast  and  very  easy  to  use  I 

Software  Organization 

a.  Training  Exgrciseg.  The  ability  to  typewrite  is  assumed,  even  by  the 
hunt  and  peck  method.  However,  certain  basic  skills  in  using  a computer  terminal 
must  be  developed  in  the  student;  use  of  the  rub-out  key,  control-X,  escape-R,  eCc. 
to  protect  him  from  being  downgraded  despite  his  recognition  of  errors  immediately 
after  typing  and  before  carriage  return;  logging-on  and  loggtng-off  procedures; 
numerical  calculating  in  the  keyboard  mode;  etc.  One  or  two  preliminary  exercises 
should  be  prepared  for  these  items . 

b.  Prime  or  Supportive  CAI 7.  This  is  always  a debatable  question.  Prime 
CAI  attempts  to  do  all  or  most  of  the  teaching  by  the  Socratic  method;  supportive 
CAI  relegates  the  teaching  to  competent  people  in  a conventional  or  unconventional 
classroom  using  live  demonstration  equipment  and  other  audio-visual  aids,  and  is 
employed  to  enrich,  exemplify,  fill-in,  drill,  test,  etc.,  the  material  provided  by 
the  teacher.  A decision  relating  to  this  question  must  be  made  in  the  planning  stage. 
Research  relating  to  training  optimization  will  assist  the  planner. 
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c.  Teacher  Training.  Aside  from  learning  to  write  programs  in  his  own  field, 
the  teacher  should  be  trained  with  respect  to  the  philosophy,  techniques,  and  stra- 
tegies which  best  apply  to  the  particular  nature  of  the  content  with  which  he  is 
working.  Only  when  the  t'.taff  or  faculty  actively  participates  in  these  tasks  can 
maximum  success  be  expected. 

d.  Format  of  Exercises . The  subconscious  reacti  jn  of  a student  to  the  format 
of  the  material  he  sees  on  the  screen  may  be  likened  to  any  reader's  reaction  to  a 
careful  published  layout  in  a textbook  or  magazine.  Just  as  publishers  strive  for 
artistic  appeal,  the  CAI  programmer  should  aim  for  proper  indentations,  double 
spaces  where  needed,  underscores,  equations  that  look  ju3t  as  they  do  in  books, 
and  so  on.  This  is  not  a fuss-pot  attitude  at  all.  The  student  will  get  the  feeling 
that  the  course  designers  have  taken  the  time  and  trouble  to  give  him  something 
appealing  rather  than  just  slapping  anything  at  all  into  the  program. 

e.  Design  for  Minimum  File  Space  Requirements . It  is  much  easier  for  the 
student  to  read  expository  material  from  a clearly  printed  sheet  of  paper  than  it 
is  to  grasp  it  as  it  prints  out  on  the  terminal  screen.  Moreover,  disc  space  de- 
pletion can  occur  very  rapidly  when  a great  deal  of  text  is  written  into  the  program. 
The  use  of  Worksheets  and  Question/Problem  Books  minimizes  both  of  these  prob- 
lems in  representing  the  state  of  the  art.  Alternatives  should  be  studied. 

f-  Software  Categories.  Supportive  CAI  software  takes  a number  of  different 
forms , both  from  the  point  of  view  of  the  motivating  philosophy  and  presentation 
styling . Categories  which  explicitly  describe  the  nature  of  the  exercises  should 
be  set  up  early  iu  the  planning.  As  examples,  categories  include  drills,  tests, 
self -diagnostic  problems,  managed-problems,  situational  simulations,  laboratory 
simulations,  etc. 

(1)  Each  of  these  groups  should  be  further  categorized  in  terms  of  func- 
tion, that  is,  is  the  exercise  in  a given  group  to  be  required  of  every  student  or 
might  it  be  optional  for  some  or  all  of  the  learners  ? 

(2)  A level-of-difficulty  rating  should  be  assigned  to  each  exercise  as 
well.  The  first  judgement  is  intuitively  based,  but  as  data  is  gathered,  these 
intuitive  classifications  may  be  readily  changed  as  required.  The  material  com- 
plement of  several  different  courses  in  the  same  field  may  then  be  properly 
adjusted  in  classifications  such  as  basic  and  advanced. 

g.  Motivational  Research  in  Student -Computer  Interaction.  This  is  one  of  the 
most  important  research  aspects  relating  to  CAI  structuring.  After  the  novelty 
wears  off,  the  computer  is  often  an  extremely  dull  lecturer ! Its  value  lies  in  its 
ability  to  elicit  student  responses,  analyze  them  for  correctness,  define  the  "hangup" 
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that  led  to  the  incorrect  response,  remediate  with  explanations,  and  finally  pre- 
scribe further  student  activity.  Simultaneiously,  records  of  the  nature  and  char- 
acteristics of  the  responses  may  be  filed  for  use  in  evaluating  performance  and 
achievement.  Some  further  advantageous  research  techniques  relating  to  and 
involving  interaction  — always  applied  in  tune  with  the  nature  of  the  subject 
matter  and  the  objective-  ^f  the  exercise  — are: 

(1)  Student  participation  in  data  selection.  In  certain  types  of  self- 
diagnostic problems  and  in  many  situational  simulations,  the  student  can  be  made 
to  feel  that  he  is  personally  immersed  in  the  activity  if  he  is  allowed  to  choose  his 
own  data  (within  carefully  stipulated  ranges)  for  the  exercise.  The  computer  then 
works  with  him  on  a one-to-one  basis,  analyzing  his  data  as  though  it  were  dedicated 
only  to  him. 

(2)  Student  participation  in  selecting  calculating  mode.  The  student  should 
always  be  permitted  to  perform  calculations  in  problems  and  laboratory  simulations 
using  his  favorite  tool.  He  may  want  to  solve  a problem  longhand  with  paper  and 
pencil,  with  arithmetical  tables  or  log  tables,  with  a portable  electronic  calculator, 
or  by  using  the  keyboard -calculator  mode  of  the  computer.  This  last  mode,  how- 
ever, does  call  for  preliminary  training  (Software  Organization,  a..  Training  Ex- 
ercises.) which  involves  learning  how  to  rephrrse  mathematical  equations  in 
acceptable  computer -input  format.  As  an  important  option,  the  keyboard  calcu- 
lation facilities  should  always  be  available  to  the  student  at  any  time  during  pro- 
gram execution. 

(3)  Student  participation  in  Socratic  development.  Normally,  the  "teacher" 
asks  a question  to  which  the  student  responds  and  is  remediated  if  necessary.  Very 
interesting  programs  can  be  prepared,  however,  in  which  the  student  must  ask 
questions  of  the  computer  and  utilize  the  answers  he  receives  in  completing  the 
exercise.  This  kind  of  challenge  is  quite  different  from  the  conventional  ones  and 
invariably  heightens  interest. 

h.  Reduction  of  Terminal  Time.  For  reasons  previously  delineated,  indivi- 
dual learner  terminal  -time- per-session  should  be  reduced  to  as  small  a practical 
figure  as  possible.  Suggestions  for  realizing  this  objective  are: 

(1)  Confine  the  exercise  to  a single  concept,  if  possible. 

(2)  Plan  the  exercise  so  that  the  student  may  prepare  for  it  on  his  own 
time,  away  from  the  terminal.  Basically,  this  means  that  he  can  prepare  drawings, 
read  schematics,  solve  equations  literally  in  anticipation  of  data  to  be  delivered 
on-line,  take  voltage  or  current  measurements,  and  in  general,  ready  himself  to 
work  quickly  and  efficiently  once  he  logs  on.  This  approach  is  especially  good  for 
problems  in  which  constrained  randomized  data  is  given  only  after  program  execu- 
tion has  begun. 
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(3)  Plan  the  exercise  so  that  the  student  can  complete  the  exercise  away 
from  the  terminal  after  the  program  has  been  run.  This  applies  particularly  to 
simulated  laboratory  exercises  in  which  a number  of  related  questions  and  problems 
are  presented  on  the  laboratory  worksheet . 

(4)  For  advanced  training  situational  simulations  involving  calculations  as 
well  as  for  problems  and  laboratory  simulations,  students  should  be  taught  how  to 
solve  an  equation  for  the  unknown  in  literal  terms . This  speeds  up  the  process  of 
keyboard  calculation  once  the  data  is  delivered  on-line.  This  is  particularly  appro- 
priate to  the  subject  material  like  the  Fort  Monmouth  course  "Communications  - 
Electronics  Systems  Engineering". 

(5)  All  of  these  suggestions  for  reduction  of  terminal  time  are  based  on  the 
meticulous  preparation  of  alternative  materials  (video,  printed  worksheets,) 

etc . , issued  to  the  student  well  in  advance  of  his  scheduled  appointment  with  the 
terminal . 

i.  Data  Files.  The  computer  files  must  be  set  up  in  a manner  so  that  a variety 
of  printed  reports  may  be  elicited  any  time,  on  derm  1 of  the  instructor  or  depart- 
ment head.  As  a minimum,  the  files  must  be  capable  of  yielding  well -formated, 
unambiguous  reports  on  things  like: 

(1)  Individual  Student  Performance  giving  figures  on  error  percentages, 
types  of  errors  (in  terms  of  needed  skills),  time-per-6ession,  response  time  for 
critical  questions  and  problems,  rate  of  progress  related  to  total  course  time,  etc. 

(2)  Individual  vs  ■ Group  Performance  giving  figures  on  rank  in  group 
based  on  rate  of  progress,  rank  in  group  based  on  grades  obtained,  comparative 
performance  in  terms  of  skill  categories  to  locate  specific  skills  which  have  not 
adequately  developed  in  specific  instances,  etc.  This  data- should  also  be  able  to 
provide  reports  for  item  analysis  of  course  material  in  order  to  validate  each  item 
objectively  so  that  the  course  may  be  edited  for  improvement,  and  so  that  level -of - 
difficulty  ratings  previously  assigned  intuitively  may  now  be  revised  empirically. 

(3)  For  both  (1)  and  (2)  above,  research  in  the  broader  field  relating  to  the 
type  of  statistics  valuable  to  the  trainer  is  indicated. 

j . Tutor  Contact . The  complete  system  should  provide  for  tutorial  contact  at 
the  time  that  it  is  needed.  A student  should  be  able  to  confer  with  his  instructor, 
peer  tutor,  etc.  before,  after,  or  while  he  is  on  line. 

k.  Reduction  of  Programming  Time.  A truly  critical  item ! 


(1)  An  effective  CAT  system  cannot  consist  of  a few  isolated  programs  to 


which  the  course  study  guide  refers  only  occasionally,  ft  must  form  an  integral, 
consistently  applied  link  in  the  chain  of  events  for  each  lesson  or  module.  This 
means  that  a relatively  large  number  of  programs  must  be  written  for  each  course 
and  in  each  category.  Herein  we  encounter  the  most  serious  obstacle  in  the  prep- 
aration of  a CAl  program.  This  problem  can  be  completely  circumvented  by  using 
the  "variant -invariant"  approach. 

(2)  hi  this  programming  procedure,  a very  carefully  structured  invariant 
section  is  written,  including  all  of  the  content  material  that  will  appear  consistently 
in  all  of  the  exercises.  About  75%  of  each  exercise  is  covered  by  the  invariant 
section.  A special  worksheet  is  then  prepared  and  reproduced  in  quantity  in  which 
only  the  line  numbers  that  need  to  be  changed  for  a specific  exercise  are  shown. 
Insertion  instructions  are  given  for  each  missing  line  so  that  the  information  may  be 
written  in  longhand.  A typist  then  inputs  this  information  and  merges  it  with  the 
variant  portion  of  the  program  right  at  the  terminal.  The  complete  program  is 
then  saved  under  its  appropriate  file  name.  For  example,  the  efficiency  realized 
by  this  method  permits  an  instructor  to  write  a new  computer -managed  problem. 


complete  with  data-randoming  instructions  and  a full  set  of  terminating  diagnostic 
questions  for  on-line  student  responses,  in  from  2 to  3 hours.  Using  the  format 
worksheet,  the  typist  can  then  input  the  information,  merge  the  programs,  check 
the  operation,  and  edit  the  program  in  less  than  2 hours! 

l.  Student  Self-Diagnosis.  All  CAI  exercises  should  contain  a set  of  diagnostic 
questions  for  use  on-line.  It  is  here  that  correction,  remediation,  and  prescription 
takes  place  in  most  categories  of  exercises.  For  the  most  part,  these  questions 
should  be  non-numerical  to  avoid  excessive  terminal  time,  but  they  must  be  expertly 
phrased  and  remediated  so  that  they  have  conceptual  value. 

m.  Software  Category  Characteristics 

(1)  Drills . randomly  selected  questions  from  an  appropriate  BANK; 
immediate  feedback  of  correct  answer  after  student  response;  records  kept  of 
responses  for  data  accumulation  only,  that  is,  student  course  grades  are  not 
affected  by  drill  results.  This  encourages  learners  to  run  through  the  drills. 
Limitation  of  the  number  of  drills  permitted  per  lesson  or  module  at  the  discre- 
tion of  the  instructor. 

(2)  Tests . resemble  drills  in  organization  except  that  the  student  is  not 
informed  of  the  correctness  of  his  answers  until  the  test  is  finished  and  he  signals 
termination.  Normally,  a weighted  grade  (based  upon  the  conglomerate  level-of- 
difficulty  ratings  for  the  questions  on  the  particular  test)  is  printed  out  on  termin- 
ation, but  the  instructor  has  the  facility  of  test  replay  for  his  or  his  student's 
edification.  All  responses  are  recorded  and  accumulated. 
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(3)  Situational  Simulations.  A situation  is  described  in  an  imaginative 
and  interest -stimulating  manner  in  the  Worksheet,  and  the  problem  is  posed.  The 
data  may  be  randomly  provided  by  the  computer  or  the  student  may  be  permitted 
to  select  his  own  data,  depending  on  the  nature  of  the  subject  matter.  The  student 
is  expected  to  show,  on  his  Worksheet,  the  relevant  equation  solved  literally  for 
the  unknown  in  both  standard  mathematical  form  and  in  computer -input  form.  Key- 
board calculation  mode  is  available. 

(4)  Laboratory  Simulations . A self-imposed  restraint  operates  here;  a 
laboratory  simulation  exercise  is  suggested  whenever  physical  conditions,  cost, 
or  non-availability  do  not  permit  it  to  be  performed  individually  by  the  student  in 
a live  laboratory!  The  computer  cannot  replace  hand-on  laboratory  work.  It  can, 
however,  provide  valuable  training  in  pre  and  post  laboratory  procedures  in  instan- 
ces where  the  equipment  and/or  supplies  that  would  be  needed  are  impractically 
bulky  or  costly.  For  example,  very  successful  laboratory  simulations  have  been 
written  based  upon  experiments  with  3 -meter  air  tracks,  particle  accelerators, 
cloud  chambers  and  bubble  chambers,  complex  organic  synthesis  in  chemistry, 
diagnostic  situations  in  medicine  and  dentistry,  electronics,  avionics,  etc. 

(5)  Computer-Managed  Problems . Generally  used  to  provide  problem 


formance  agenda  items"  should  be  included: 

(1)  The  model  should  "remember"  student's  performance  on  previous 
activities  requiring  similar  skills,  in  order  to  guide  and  select  strategies. 

(2)  Interface  system  with  student's  instrumentation  to  permit  analog-to- 
digital  input  into  ABACUS,  with  consequent  guidance  provided  by  the  system. 

(3)  Extend  above  concept  to  the  point  of  full  failure  simulation,  with 
computer -controlled  guidance  toward  problem  solution. 
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(4)  Record  rate -of -progress  data,  so  as  to  detect  unusual  changes  in 
student  performance  due  to  curriculum  mis -design,  or  due  to  student  mis- 
understanding. 


(5)  Analyze  types  of  skills  required  for  successful  achievement  on  each 
test  (particularly  the  "hands-on"  evaluations).  Record  achievement  level  for  each 
skill  type,  during  evaluation.  Correlate  with  other  tests  of  that  skill  type  for  the 
individual  student,  and  with  the  performance  of  other  students  on  that  test. 

(6)  Record  and  correlate  student  learning  style  characteristics,  across 
course  boundaries.  Incorporate  data  from  initial  military  test  battery,  and  use 
course  evaluative  information  to  update  data  from  battery. 


(7)  Design  on  overall  educational  management  information,  system  which 
incorporates  all  of  student's  test  information  as  well  as  personneVlaata.  Should 
facilitate  curriculum  redesign  process . 


q.  Overall  Summary.  In  overall  summary,  the  larger  research  areas  deal  with 
with  motivation,  better  hardware,  better  software,  better  language,  better  simu- 
lation, and  overall  techniques  designed  to  optimize  individual  learning  experiences 
for  the  military  student  with  reduced  training  time  and  costs . 
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Introduction 

a.  The  ABACUS  program  is  clearly  shifting  from  a CAI  to  a CMI  emphasis. 

This  is  creating  a need  to  re-examine  the  R&D  program  to  make  it  consistent  with 
the  new  emphasis.  This  paper  is  an  attempt  to  look  at  one  problem  in  this  context, 
namely  a part  of  the  formative  evaluation  process.  This  paper  provides  a meta- 
model for  use  in  the  CMI  application  contemplated  at  Fort  Gordon.  It  builds  upon 
the  authors  previous  work  with  the  U.  S.  Naval  Academy  and  provides  workable 
algorithms  for  processing  data  from  student  performance.  The  processes  of  data 
analysis  are  designed  to  support  a decision  maker  who  is  responsible  for  the 
quality  of  the  instructional  program.  The  procedures  are  described  at  a level  which 
will  permit  them  to  be  used  directly  in  a variety  of  contexts.  They  are  not  just  for 
a particular  training  program. 

b.  The  rationale  for  this  paper  is  that  it  consolidates,  in  a decision  context  and 
in  a more  rigorous  form  than  is  now  available,  the  processes  that  provide  the  bases 
for  changing  a system  of  instruction.  By  systematizing  these  processes  and  building 
them  into  a program,  a process  of  iteration  and  validation  can  be  started,  and  it  can 
converge  on  a level  of  quality  specified  by  the  system  managers.  A CMI  software 
effort  could  be  developed  to  use  CTS  as  the  primary  agent  in  this  process.  This  will 
require  programming  and  further  development  and  testing  of  the  concepts  and  de- 
cision processes  included  here. 

Background 

a.  During  the  past  decade  evaluators  of  programmed  instruction  and  computer 
aided  instruction  have  recognized  that  it  is  very  difficult,  if  not  impossible,  to 
determine  subjectively  the  effectiveness  of  test  items  and  instruction.  (See  Rothkopf, 
1963.)  In  this  paper  we  specify  a set  of  objectives  rules,  based  upon  item  perform- 
ance data,  for  identifying  those  test  items  and  sections  of  instruction  that  seem  to 
require  revision.  This  objective  method  should  provide  a more  rational  basis  for 
decision-making  than  the  subjective  method  of  making  decisions  based  primarily 
upon  some  unidentified  combination  of  subject  matter  knowledge,  experience,  and 
intuition. 

b.  The  rationale  for  decision-making  that  we  propose  is  basically  an  elaboration 
of  a technique  devised  by  Stolurow  and  Frase  (1968).  Their  method  is  based  upon  a 
comparison  cf  three  different  types  of  error  rates  for  program  frames:  (a)  the 
theoretical  error  rate,  which  is  the  error  rate  expected  simply  on  the  basis  of 
random  guessing;  (b)  the  base  error  rate,  which  is  the  error  rate  obtained  by  stu- 
dents not  exposed  to  the  teaching  material  for  the  frame;  and  (c)  the  instructional 
error  rate,  which  is  the  error  rate  obtained  by  students  who  have  been  exposed  to 
the  instruction. 
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c.  In  this  paper  we  will  treat  not  only  program  frames  (i.e. , test  items)  that 
are  an  integral  part  of  instruction,  but  also  test  items  that  occur  both  before  and 
after  instruction.  In  addition,  we  will  use  both  error  rates  and  discrimination  in- 
dices as  data  for  decision  making. 

d.  In  order  to  put  the  proposed  decision  process  into  a conceptual  context,  let  us 
assume  that  we  have  an  instructional  program  teaching  a set  of  terminal  objectives. 
Chronologically,  each  terminal  objective  is  tested  by  (a)  a pretest  item  that  occurs 

I before  the  objective  has  been  taught,  (b)  a terminal  test  item  that  occurs  almost 

immediately  after  the  objective  has  been  taught,  and  (c)  a posttest  item  that  occurs 
"some  time  after"  the  objective  has  been  taught.  Without  loss  of  generality,  we  will 
assume  (as  is  usually  the  case)  that  the  set  of  pretest  and  posttest  items  form  two 
tests  that  occur,  respectively,  prior  to  and  following  the  instruction  for  all  objec- 
tives. Furthermore,  we  will  assume  that  all  of  the  items  testing  any  objective  are 
either  identical  or  "corresponding".  (The  concept  of  "corresponding"  items  will  be 
treated  in  detail  later;  however,  we  can  roughly  define  corresponding  items  as  items 
that  test  the  same  objective  at  the  same  level  of  difficulty.)  Given  this  conceptual 
context,  it  is  clear  that  most  forms  of  individualized  instruction  (including  PI,  CAI, 
and  multi-media  modes  of  instruction)  can,  and  often  do,  meet  these  minimum 
requirements . 

e.  In  the  final  analysis,  using  item  performance  data,  we  want  to  identify  those 
test  items  and  sections  of  instruction  (relevant  to  a given  objective)  that  require  re- 
vision. The  decision  process  we  propose  will  not  necessarily  tell  the  evaluator  how 
to  revise  items  and/or  instruction,  but  the  process  will  provide  objective  rules  for 
deciding  what  to  revise. 


Types  of  Data  and  Decisions 


Error  rate  is  defined  as  the  proportion  of  students  getting  an  item  incorrect, 

i.e. , 


Error  Rate  = Number  of  Incorrect  Answers  (1) 

Total  Number  of  Answers 


or 


(2) 
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where  ERj  means  error  rate  for  item  i,  N is  the  total  number  of  students  answering 
the  item,  Xjj  = 1 if  student  j gets  item  1 correct,  and  Xjj  = 0 if  student J gets  item 
i wrong.  We  can  also  express  Equation  2 as : 


N 


i = i 


XU 


(3) 


ERi  = 1 - 


N 


Since  the  last  term  on  the  right  of  Equation  3 is  item  difficulty  level  (DLj),  it  is 
clear  that 


ERi  = 1 - DLi 


(4) 


Equation  4 shows  that  from  a theoretical  viewpoint  it  is  immaterial  whether  we  use 
difficulty  level  or  error  rate;  however,  in  our  opinion,  using  error  rate  seems  to 
facilitate  an  understanding  of  some  of  the  decisions  that  will  be  proposed  later. 


f.  For  many  of  the  proposed  decision  rules  we  will  assume  that  error  rates  are 
classified  as  either  high  (H)  or  low  (L),  and  that  the  evaluator  predetermines  an 
appropriate  cut-off  point  between  high  and  low  error  rate.  For  any  given  objective, 
the  cut-offs  for  the  error  rates  discussed  below  must  be  identical  in  order  to  apply 
the  rules  that  will  be  specified.  Also,  in  most  cases,  the  cut-offs  chosen  will  prob- 
ably be  the  same  for  all  objectives;  however,  occasions  can  arise  when  certain 
objectives  should  have  a higher  (or  lower)  error  rate  out -off  than  other  objectives. 
For  example,  items  testing  very  crucial  objectives  might  be  assigned  a cut-off  of 
0.10,  while  other  items  might  have  a cut-off  of  0.25. 


g.  Discrimination  indices  will  be  classified  as  either  positive  (+),  negative  (-), 
or  nondiscriminating  (0).  By  positive  and  negative  indices  we  mean  indices  that 
discriminate  significantly  (as  some  appropriate  OC  level)  in  the  positive  and  nega- 
tive directions,  respectively.  The  discrimination  index  used  should,  of  course,  be 
appropriate  for  the  data  in  question. 


h.  Before  instruction  we  can  obtain  three  types  of  data  for  each  objective  that 
has  a pretest  item: 


(1)  the  Theoretical  Error  Rate  (TER),  which  is  the  expected  proportion  of 
students  getting  a pretest  item  incorrect  simply  on  the  basis  of  random  guessing; 
l.e. , if  K is  the  number  of  possible  answers  to  an  item,  then 


TER  = K - 1 
K 


(5) 
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For  example,  if  an  item  has  five  alternatives,  we  would  expect  80  percent  of  the 
students  to  get  the  item  incorrect  simply  by  guessing,  without  any  knowledge  of  the 
objective  tested  by  the  item; l.  2 

(2)  the  Base  Error  Rate  (BER),  which  is  the  observed  proportion  of  students 
getting  a pretest  item  incorrect;  aid 

(3)  the  Base  Discrimination  Index  (BDI),  which  is  the  discrimination  index 
for  a pretest  item.  (We  will  use  total  score  on  the  pretest  as  the  criterion  variable 
for  BDI.) 

i.  After  instruction  we  can  obtain  two  types  of  data  for  each  objective  that  has  a 
posttest  item:  (a)  the  Posttest  Error  Rate  (PER)  and  (b)  the  Posttest  Discrimina- 
tion Index  (PDI).  (Total  score  on  the  posttest  will  be  used  as  the  criterion  variable 
for  PDI.) 

j . Immediately  following  the  instruction  for  any  objective  we  can  obtain  the 
Instructional  Error  Rate  (IER),  which  is  the  error  rate  on  a terminal  test  item  for 
a given  objective.  3 Note  that  IER  refers  to  the  error  rate  on  a terminal  item,  not 
the  error  rate  on  other  questions  associated  with  teaching  the  given  objective.  We 
will  not  consider  Instructional  Discrimination  Index  since,  in  our  opinion,  it  does 
not  seem  to  be  very  useful  for  making  decisiors  beyond  those  that  can  be  made  with 
the  other  types  of  data. 

k.  In  subsequent  sections  we  will  analyze  the  decisions  that  can  be  made  on  the 
basis  of:  (a)  pretest  data,  alone;  (b)  posttest  data,  alone;  (c)  pretest  and  posttest 
data;  and  (d)  pretest,  posttest,  and  terminal  item  data,  hi  this  way,  the  cumulative 
contribution  of  these  types  of  data  to  the  decision  process  should  be  evident.  Then 
we  will  discuss  the  decisions  that  can  be  made  based  upon  arithmetic  differences 
between  various  error  rates.  Subsequently,  two  sets  of  data  will  be  discussed:  the 
first  will  be  used  to  empirically  demonstrate  the  contribution  of  the  various  types  of 
data  to  the  decision  process  and  the  second  will  be  used  to  present  an  efficient  ap  - 
plication of  the  decision  process . 

l.  For  each  decision  rule  presented  we  will  give  our  reasons  for  specifying 
whether  test  items  or  instruction  relevant  to  a given  objective  should  be  revised  (R), 
questioned  (?),  or  not  revised  (NR).  These  decisions  should  not,  however,  be 
interpreted  too  strictly;  the  evaluator  will  still  have  to  use  some  degree  of  subjective 

judgment.  For  example,  when  we  say,  in  subsequent  discussions,  that  an  item  should 
be  revised  (R),  we  mean  that  our  best  guess  on  the  basis  of  the  data  is  that  the  item 
should  be  revised,  but  the  evaluator  must  make  the  final  decision.  Also,  when  we 

say  that  an  item  (or  instruction)  is  questionable  (?),  we  mean  that  the  data  are  not 
sufficient  to  make  a definite  judgment  about  whether  or  not  the  item  (or  instruction) 
should  be  revised. 
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m.  Since  we  are  assuming  that  all  items  testing  a given  objective  are  Identical 
or  "corresponding",  a decision  about  item  revision  applies  equally  to  all  items  test- 
ing the  objective  in  question.  Thus,  when  we  say  that  an  item  should  be  revised,  we 
mean  that  our  best  guess  is  that  all  items  testing  the  given  objective  should  be  re- 
vised. On  the  other  hand,  when  we  say  that  instruction  should  be  revised,  we  mean 
that  that  part  of  the  instructional  system  that  attempts  to  teach  the  given  objective 
should  be  revised. 

n.  One  additional  consideration  deserves  mention.  Ideally,  one  would  validate 
his  test  items  prior  to  using  them  in  an  instructional  system;  however,  this  is  often 
not  feasible,  especially  in  the  criterion-referenced  testing  situation  that  is  an  integral 
part  of  many  instructional  systems.  Therefore,  in  most  cases,  the  activity  of  forma- 
tive evaluation  must  take  into  account  the  possible  invalidity  of  both  test  items  and 
instruction.  For  this  reason,  most  of  the  decision  rules  that  will  be  presented  arv 
based  upon  the  assumption  that  we  have  no  a priori  reason  to  believe  that  test  items 
are  more  valid  than  instruction  or  vice-versa. 

Pretest  Data 

a.  Prior  to  instruction  we  can  collect  three  sets  of  data:  Theoretical  Error 
Rate  (TER),  Base  Error  Rate  (BER),  and  Base  Discrimination  Index  (BDI).  Given 
these  three  sets  of  data,  various  reasonable  rules  can  be  formulated  for  making  de- 
cisions about  whether  or  not  to  revise  test  items,  ft  is  not  likely  that  only  pretest 
data  would  be  used  to  make  decisions  about  items,  yet  it  is  useful  to  consider  the 
types  of  decisions  that  are  appropriate  on  the  basis  of  such  data. 

(1)  Rule  1.  If  TER  and  BER  are  both  the  same  (i.e. , H,  H or  L,  L),  then 
no  necessity  for  revision  is  indicated.  In  this  case,  the  observed  error  rate  (BER) 
without  benefit  of  instruction  is  approximately  the  same  as  the  expected  error  rate 
(TER). 

(2)  Rule  2.  If  TER  is  low  (L)  and  BER  is  high  (H),  then  no  revision  is  in- 
dicated. This  anomalous  case  could  arise  if  the  particular  objective  for  the  item 
involved  concepts  that  are  typically  misunderstood.  For  example,  many  students 
(in  the  authors'  opinion)  believe  that  "inflammable"  and  "flammable"  have  different 
meanings.  If  an  item  were  constructed  testing  whether  or  not  'Inflammable"  and 
"flammable"  have  the  same  meaning,  and  if  this  item  were  given  prior  to  instruc- 
tion, it  is  quite  possible  that  more  students  would  get  the  item  incorrect  than  we 
would  expect  on  the  basis  of  the  theoretical  error  rate  (TER).  In  this  case,  there 
is  no  reason  to  revise  the  item;  rather,  we  expect  that  the  instruction  will  correct 
the  students'  misconception.  This  should  be  validated  within  the  training  situation. 

(3)  Rule  3.  If  TER  is  high  (H),  and  BER  is  low  (L),  then  the  item  will  prob- 
ably need  to  be  revised.  In  this  case,  students,  without  benefit  of  instruction,  are 
performing  considerably  better  than  expected.  It  appears  that  the  item  itself  may 
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be  teaching  or  that  one  or  more  distractors  are  so  easy  that  many  students  can  pick 
the  correct  answer  largely  by  a process  of  elimination.  Another  possible  explana- 
tion is  that  the  readability  level  of  certain  distractors  is  so  high  that  the  test -wise 
student  correctly  guesses  that  they  are  incorrect.  (See  Klare,  1963,  for  techniques 
to  determine  readability  level.)  In  any  case,  the  item  seems  to  need  revision.4 

(4)  Rule  4.  If  an  item  is  negatively  discriminating  before  instruction,  then 
the  item  is  questionable  in  that  it  may  need  revision.  IT,  however,  the  item  is  posi- 
tively discriminating  or  nondiscriminating,  then  no  revision  is  indicated.  A nega- 
tively discriminating  item  is  questionable  since  it  indicates  that  the  poorer  students 
(on  the  basis  of  total  test  score)  are  out-performing  the  better  students;  however,  a 
situation  similar  to  that  indicated  in  Rule  2 could  be  the  cause  of  the  negative  dis- 
crimination index.  A positively  discriminating  item  is  quite  possible  and  reasonable 
prior  to  instruction  simply  because  some  good  students  are  usually  expected  to  per- 
form better  than  chance  on  a pretest.  A nondiscriminating  item  is  the  best  of  all 
possibilities . 

(5)  Rule  5.  If  an  item  is  positively  or  negatively  discriminating  before  in- 
struction, then  the  prerequisites  for  the  objective  tested  by  the  item  should  be 
checked.  Clearly,  whenever  an  item  is  discriminating  (either  positively  or 
negatively)  one  group  (upper  or  lower)  is  outperforming  the  other  group  (lower  or 
upper).  In  such  a case,  it  seems  reasonable  to  check  whether  or  not  the  group  with 
the  higher  error  rate  does,  in  fact,  possess  the  prerequisites  necessary  to  achieve 
the  given  objective. 

b.  These  rules,  as  well  as  all  other  rules  that  will  be  discussed,  are  given  in 
abbreviated  form  in  Table  1. 
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32*1 


H H 

L L 


L H 

H L 


TABLE  1.  Rules  for  Decision-Making  (Cont) 


Error  Rates 

Decision1 

Rule 

No. 

TER  BER  EDI 

IER  PER  PDI 

Item 

Instruction  Prerequisites 

12 

L L 

NR 

13 

L H 

R 

14 

H L 

? 

15 

H H 

R 

16 

DER*5 

R 

DER(NS)C 

NR 

17 

RER>  Cj 

K 

0<  RERl.C! 

NR 

18 

RER  < -c2 

? 

-c2<  RER<  0 

NR 

19 

PMPG  < c3 

R 

PMPG  » C3 

NR 

a"NR"  means  no  revision  required. 

"R"  means  revision  is  required. 

means  the  data  are  not  sufficient  to  make  a sound  judgment  about 
whether  or  not  revision  is  required. 

"E"  means  the  prerequisites  for  the  objective  should  be  examined. 


t 

I 

i 
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^DER  is  significantly  greater  than  zero  at  the  . 05  level  for  a one-tailed  test  of 
significance. 

CDER  is  not  significantly  greater  than  zero  at  the  . 05  level  for  a one-tailed  test 
of  significance. 

Note. — ci,  C2,  C3  are  cut-offs  chosen  by  the  evaluator. 


Posttest  Data 


As  a result  of  administering  a posttest  two  types  of  data  can  be  collected:  the 
Posttest  Error  Rate  (PER)  and  the  Posttest  Discrimination  Index  (PDI).  Since  these 
data  are  collected  alter  instruction,  theoretically  decisions  can  be  made  aboui  both 
items  and  instruction;  however,  it  is  very  difficult  to  identify  items  and  instruction 
that  should  be  revised  solely  on  the  basis  of  posttest  data.  In  almost  every  case,  we 
can  say  whether  or  not  there  is  something  wrong,  but  we  cannot  pinpoint  the  problems . 

a.  Rule  6.  If  PER  = L and  PDI  = 0,  then  neither  the  item  nor  the  instruction 
need  to  be  revised.  This  is  the  best  possible  situation,  since  the  optimal  conditions 
for  both  error  rate  and  discrimination  index  are  fulfilled;  i.e. , at  the  end  of  instruc- 
tion we  hope  that  most  of  the  students  get  the  posttest  item  correct  (PER  ■ L),  and 
that  the  item  is  nondiscriminating  (PER  = 0).  (Later  we  will  discuss  our  reasons  for 
preferring  nondiscriminating  items.) 

b.  Rule  7.  If  PER  = L and  PDI  = + or  -,  then  both  the  item  and  the  instruction 
are  questionable.  The  fact  that  PDI  is  clearly  nonzero  indicates  a possible  need  for 
revision. 

c . Rule  8.  If  PER  = H and  PDI  = -,  then  both  the  item  and  instruction  should  be 
revised,  since  PER  = H and  PDI  = - is  the  worst  possible  situation  that  can  occur. 

It  is  possible  that  either  the  item  or  the  instruction  is  at  fault,  but  not  both;  however, 
we  assume  here  that  the  most  universally  applicable  decision  is  to  check  both  the 
item  and  the  instruction  to  see  what  revisions  are  needed. 

d.  Rule  9.  If  PER  = H and  PDI  = + or  0,  then  the  instruction  should  be  revised 
and  the  item  should  be  questioned.  Whenever  error  rate  is  high  after  instruction, 
something  is  wrong,  but  without  additional  information  we  do  not  know  whether  the 
fault  definitely  lies  with  the  item  or  the  instruction.  However,  the  authors  believe 
that  evaluators  are  often  more  confident  about  test  items  than  they  are  about  methods 
or  materials  for  instruction;  it  is  also  possible  that  the  test  items  have  been  pre- 
viously validated  or  partially  validated.  Therefore,  in  this  case,  it  seems  reason- 
able to  place  a less  stringent  criterion  on  the  item  than  on  the  instruction.  It  should 
be  noted,  however,  that  perceptions  can  be  biased;  i.e. , the  test  item  could  be  at 
fault.  It  is  certainly  advisable  to  analyze  the  nature  of  any  validation  or  pre- 
validation activity  for  its  applicability  in  the  present  context  since  sampling,  testing, 
and  teaching  conditions  can  vary  considerably. 

e.  Rule  10.  When  PDI  = + cr  PDI  = -,  then  the  prerequisites  for  the  objective 
tested  by  the  item  should  be  examined.  The  reason  for  this  decision  is  identical  to 
that  presented  in  Rule  5 in  the  previous  section. 


Pretest  and  Posttest  Data 


a.  It  is  evident  from  Table  1 that  neither  the  pretest  data  alone  (See  Rules  1-5.) 
nor  the  posttest  data  alone  (See  Rules  6-10.)  give  the  evaluator  much  indication  about 
which  items  and/or  sections  of  instruction  should  be  revised.  Clearly,  more  mean- 
ingful decisions  can  be  made  by  combining  the  two  sets  of  data.  When  this  is  done, 
all  of  the  rules  discussed  in  the  last  two  sections  are  applicable,  with  the  exception 
of  Rule  5 which  is  superseded  by  Rule  10.  In  addition,  one  more  rule  can  be 
specified. 

b.  Rule  11.  If  BDI  = - and  PDI  = -,  then  the  item  should  be  revised.  Both  be- 
fore and  after  instruction  the  item  is  negatively  discriminating,  which  means  that  the 
upper  group  (based  on  total  test  score)  has  a proportionately  higher  error  rate  than 
the  lower  group.  This  clearly  is  an  unfortunate  circumstance  indicating  that  the  item 
should  be  revised. 

Pretest.  Posttest , and  Terminal  Item  Data 

a.  Recall  that  Instructional  Error  Rate  (IER)  is  the  error  rate  on  a terminal 
item  immediately  following  instruction.  If,  in  addition  to  pretest  and  posttest  data, 
we  also  take  into  account  IER,  it  is  possible  to  make  fairly  definite  statements  about 
whether  or  not  to  revise  most  segments  of  instruction  that  are  related  to  terminal 
objectives.  The  addition  of  IER  does  not,  however,  tell  us  much  more  about  the  re- 
vision of  items  than  we  already  know  from  pretest  and  posttest  data.  All  of  the  rules 
previously  specified  are  applicable  except  for  Rule  5 which  is  superseded  by  Rule  10. 
Also,  we  can  specify  four  additional  rules. 

(1)  Rule  12.  If  instructional  Error  Rate  (IER)  and  Posttest  Error  Rate  (PER) 
are  low,  then  no  revision  (NR)  of  instruction  is  indicated.  Both  during  instruction 
and  after  instruction  most  of  the  students  seem  to  achieve  the  objective  (tested  by  the 
instructional  item  and  the  posttest  item);  therefore,  we  have  two  indications  that  the 
instruction  is  adequate,  and  no  revision  is  indicated. 

(2)  Rule  13.  If  IER  = L and  PER  = H,  then  the  instruction  should  be  revised. 
During  instruction,  students  seem  to  achieve  the  objective,  but  on  the  posttest  the 
same  students  have  a higher  error  rate  for  the  same  objective.  Thus  the  data  in- 
dicate a retention  problem,  and  the  instruction  should  be  revised  to  correct  this 
situation.  Perhaps  more  practice  or  review  is  needed.  Factors  known  to  affect 
short-term  retention  should  be  considered  in  making  revisions. 

(3)  Rule  14.  If  IER  = H and  PER  = L,  then  the  instruction  should  be  ques- 
tioned. This  is  probably  an  unlikely  situation  that  would  seldom  occur  in  practice, 
although  it  may  be  explainable  in  terms  of  reminiscence  (See  McGeoch  and  Irion, 

1952;  Estes,  1970.)  and  interference  (negative  transfer).  In  any  case,  the  fact  that 
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students  experience  a high  error  rate  on  a terminal  test  item  (frame  in  CAl)  during 
instruction  seems  to  indicate  that  something  may  be  wrong  with  the  instruction.^ 

(4)  Rule  15.  If  IER  = H and  PER  = H,  then  the  instruction  definitely  should  be 
revised.  Both  during  and  after  instruction  students  do  not  seem  to  achieve  the  ob- 
jective under  consideration.  We,  therefore,  have  two  indications  of  a need  for  re- 
vising the  instruction. 

b.  Note  that  in  the  context  of  Rules  12-15,  a posttest  item  is,  in  part,  a measure 
of  retention.  Clearly,  the  evaluator  must  temper  his  decisions  about  revision  with 
knowledge  about  the  length  of  time  intervening  between  instruction  and  testing,  and 
especially  about  the  type  of  intervening  activity.  The  criticality  of  forgetting  and  the 
processes  determining  it  should  be  carefully  considered.  (See  Adams,  1967.) 

Decisions  Based  Upon  Differences  Between  Error  Rates 

a.  The  utility  of  most  of  the  foregoing  decision  rules  is  dependent  upon  the 
evaluator's  choice  of  a cut-off  level  differentiating  high  and  low  error  rates.  Dichot- 
omizing error  rate  in  this  way  clearly  facilitates  the  identification  of  appropriate 
decision  rules,  and,  in  many  cases,  the  simplicity  of  the  technique  will  probably 
outweigh  any  loss  of  precision.  However,  we  can  also  specify  an  additional  set  of 
four  useful  decisions  rules  that  take  into  account  quantitative  differences  between 
error  rates.  Three  of  these  rules  increase  the  power  or  previous  decisions,  the 
other  provides  essentially  new  information.  We  will  call  these  error  rates  "derived" 
error  rates  in  order  to  distinguish  them  from  the  "raw"  error  rates  discussed  in  the 
previous  sections. 

b.  Let  us  consider  several  limits  and  limitations  of  the  high/low  classification 
procedure  for  error  rates.  Suppose  that  Theoretical  Error  Rate  (TER)  and  Base 
Error  Rate  (BER)  for  a given  objective  are  both  classified  as  high  (H),  while  Instruc- 
tional Error  Rate  (IER)  and  Posttest  Error  Rate  (PER)  are  both  classified  as  low  (L). 
Clearly,  any  actual  arithmetic  differences  between  TER  and  BER,  as  well  as  between 
IER  and  PER,  will  not  affect  the  decisions  we  have  thus  far  proposed.  Also,  since 
BER  and  IER  are  merely  classified  as  high  and  low,  respectively,  we  will  not  have  a 
quantitative  measure  of  how  much  learning  has  actually  taken  place. 

Difference  Error  Rate 


a.  Rules  1-3  are  useful  for  making  decisions  based  upon  categorical  differences 
between  BER  and  TER,  but  we  can  make  more  accurate  decisions  by  actually  com- 
puting the  differences  between  these  error  rates . 

Let: 


DER  = TER  - BER, 


(6) 
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where  DER  stands  for  "Difference  Error  Rate".  If  DER  = 0,  then  the  observed  error 
rate  (BER)  on  the  pretest  item  in  question  is  identical  to  the  expected  error  rate 
(TER).  If  DER  < 0,  then  fewer  students  are  getting  the  item  correct  than  we  would 
expect  on  the  basis  of  random  guessing.  Finally,  if  DER  > 0,  then  more  students 
are  getting  the  item  correct  than  we  would  expect.  As  discussed  previously,  the  last 
possibility  is  often  an  unfavorable  situation,  since  it  can  mean  that  the  item  somehow 
"gives  away"  the  correct  answer. 

b . We  can  test  the  significance  of  a positive  difference  between  BER  and  TER  by 
computing 


DER  - 1/2(N) 


where  N is  the  total  number  of  students  in  the  sample.  (See  Snedecor  & Cochran, 
1967,  p.  210.) 6 The  computer  Z value  is  then  compared  with  the  normal  curve 
standard  score  at  an  appropriate  level  of  significance  for  a one-tailed  test.  (Note 
that  we  are  interested  only  in  positive  values  of  DER. ) We  can  now  specify  a more 
precise  rule  to  replace  Rules  1-3. 

c.  Rule  16.  If  the  value  of  DER  is  significantly  greater  than  zero,  then  the  item 
should  be  revised.  In  all  other  cases  no  revision  is  required. 

Retention  Error  Rate 

a.  Rules  12-15  are  useful  foi  leaking  decisions  based  upon  categorical  differ- 
ences between  IER  and  PER,  but  we  can  supplement  these  decisions  by  calculating 
the  actual  difference  between  IER  and  PER  and  comparing  this  value  to  some  pre- 
assigned cut-off.  Let: 

RER  = PER  - IER, 

where  RER  stands  for  'Retention  Error  Rate".  If  RER  = 0,  then  the  number  of 
errors  on  the  posttest  item  and  the  related  terminal  item  is  identical,  and  no  re- 
tention problem  is  evident.  If  RER  > 0,  then  students  make  more  errors  on  the 
posttest  item  than  on  the  terminal  item.  The  latter  situation  can  be  serious  if  RER 
is  considerably  greater  than  zero;  however,  it  is  not  clear  how  to  define  "consider- 
ably greater  than  zero". 

b.  We  could,  of  course,  statistically  test  whether  or  not  RER  is  significantly 
greater  than  zero  if  certain  distributional  assumptions  can  be  made,  but  such  a test 
would  not,  in  our  opinion,  provide  a meaningful  basis  for  decision.  What  is  needed 
is  a cut-off  above  which  the  amount  of  forgetting  is  great  enough  to  justify  revision 
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of  instruction.  Such  a cut-off  must  take  into  account  the  criticality  of  forgetting 
. which  in  turn  is  dependent  upon  many  factors  including  the  subject  matter  of  the  in- 
structional system  and  the  student  population  for  which  the  system  is  being  developed. 
Furthermore,  there  is  no  theoretical  rationale  for  specifying  the  same  cut-off  for  all 
items.  Thus,  in  our  opinion  only  the  evaluator  can  make  an  appropriate  choice  of  a 
useful  cut-off.  7 It,  therefore,  seems  reasonable  to  specify  the  following  rule  as  a 
more  powerful  version  of  Rule  13 . 

c.  Rule  17.  If  RER  > cj,  where  c^  is  a cut-off  specified  by  the  evaluator,  then 
the  instruction  should  be  revised,  since  the  data  indicate  a retention  problem.  If 

0 RER  < ci,  then  no  revision  is  required.  The  cut-off,  ci,  need  not  be  the 
same  for  all  objectives. 

d.  The  one  possibility  that  we  did  not  consider  above  is  RER  0;  i.e.,  students 
make  fewer  errors  on  the  posttest  item  than  on  the  terminal  item.  We  stated  pre- 
viously, in  the  discussion  of  Rule  14  that  this  is  an  unlikely  occurrence;  however, 
the  evaluator  may  want  to  specify  a cut-off  below  which  he  considers  this  problem  to 
be  serious  enough  to  merit  a closer  examination  of  the  instruction.  Again,  research 
findings  concerning  reminiscence  and  negative  transfer  may  be  useful  in  guiding 
revision. 

e.  Rule  18.  If  RER  < - c2,  is  a cut-off  specified  by  the  evaluator,  then  the 
instruction  should  be  questioned.  If  -c2  ^ RER  < 0,  then  no  revision  is  required. 
As  before,  the  cut-off  c2  need  not  be  the  same  for  all  objectives.  (See  footnote  5.) 

Percentage  of  Maximum  Possible  Gain 

a.  None  of  the  decisions  discussed  up  to  this  point  has  made  use  of  any  measure 
of  gain  in  knowledge  relevant  to  a given  objective  that  results  from  the  instructional 
system.  It  is  probably  true  that  gain  is  not  as  important  as  final  performance  on  the 
posttest,  in  most  instructional  systems;  however,  if  students  experience  relatively 
little  gain  as  a result  of  experiencing  instruction,  one  can  legitimately  question  the 
value  of  the  instructional  system  itself.  Thus,  measures  of  gain  have  long  been  a 
subject  of  considerable  interest  in  the  fields  of  programmed  instruction,  computer- 
aided  instruction,  and  multimedia  instruction.  (See  Lumsdaine,  1965.) 

% 

b.  The  simplest  measure  of  gain  for  an  objective  is  the  difference  between  theo- 
retical error  rate  (TER)  and  error  rate  on  the  corresponding  terminal  item  (IER); 
however,  a better  measure  is  the  difference  between  error  rate  on  a pretest  item 
(BER)  and  error  rate  on  the  corresponding  terminal  item  (IER).  8 The  latter  meas- 
ure would,  however,  mean  that  a gain  of  0.50  resulting  from  BER  = 1.00  and  IER  = 
0.50  would  be  indistinguishable  from  a gain  of  the  same  magnitude  resulting  from 
BER  = 0.50  and  IER  = 0.00.  In  the  former  case,  the  instructional  system  has  failed 
to  produce  50  percent  of  the  gain  in  performance  that  could  be  achieved,  while  in  the 
latter  case,  the  instructional  system  has  produced  as  much  gain  as  possible  given 
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the  entry  level  of  the  students.  Thus,  in  the  former  case,  some  revision  of  the  in- 
struction may  be  desirable,  while  in  the  latter  case,  no  revision  in  the  instructional 
system  is  required  on  the  basis  of  these  data. 


c.  The  above,  rather  trivial  example,  illustrates  that  simple  gain  does  not  pro- 
vide a very  meaningful  basis  for  revising  instruction.  A better  measure  is  percent 
of  maximum  possible  gain  for  an  objective,  defined  as: 

BER  - IER 

PMPG  = /Q, 


BER 

In  order  to  make  use  of  this  measure  the  evaluator  must  specify  a cut-off  that  deter- 
mines whether  or  not  a given  value  for  PMPG  indicates  a need  for  revision. 

d.  Rule  19.  If  PMPG  < C3,  where  C3  is  a cut-off  specified  by  the  evaluator, 
then  the  instruction  should  be  revised.  The  cut-off  c3  need  not  be  the  same  for  all 
objectives.  If  PMPG  ^ C3,  then  no  revision  is  required. 

e.  The  literature  contains  many  in-depth  discussions  and  debates  about  the  prob- 
lems and  pit -falls  associated  with  measures  of  gain.  (See,  for  example,  Cronbach 
and  Furby,  1970;  DuBois,  1962;  and  Harris,  1963.)  Most  of  this  literature,  however, 
treats  measures  of  gain  in  the  context  of  their  use  in  inferential  statistics  or  corre- 
lational analysis.  While  we  appreciate  the  importance  of  these  issues,  we  hasten  to 
add  that  measures  of  gain,  merely  as  descriptive  statistics,  can  provide  useful  in- 
formation to  evaluators.  We  believe  that  the  use  of  PMPG,  as  data  for  evaluation 
purposes,  is  a case  in  point. 

f.  When  data  of  the  type  discussed  in  this  section  are  used  along  with  the  basic 
pretest,  posttest,  and  terminal  item  data,  then  the  appropriate  decision  rules  are: 
6-11  and  16-19.  If  only  pretest  and  posttest  data  are  available,  then  Rule  16  can  be 
used  to  replace  Rules  1-3. 

Two  Examples 

a.  The  data  reported  in  Table  2 are  based  upon  the  responses  of  28  students  to 
a subset  of  test  questions  (both  completion  and  multiple -choice  varieties)  in  an  inter- 
active CAI  program  in  microeconomics.9 
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Derived 


f 

f 


Raw  Error  Rates  and  PiBcrlml nation  Indices Error  Rates 

Objective 


Number 

TER 

BER 

BDI 

IER 

PER 

PDI 

DER 

RER 

PMPG 

1 

1.000 

.750 

.365 

.250 

.071 

.205 

.250* 

-.179 

.667 

2 

.875 

.964 

.304 

.143 

.643 

.880** 

-.089 

.500 

.852 

3 

.750 

.786 

.055 

.107 

.106 

.205 

-.036 

-.001 

.864 

4 

.750 

.714 

.650** 

.214 

.036 

.141 

.036 

-.178 

.700 

5 

.500 

.604 

. 786** 

.321 

.000 

.000 

-.104 

-.321 

.469 

6 

.667 

.714 

.475* 

.107 

.179 

.015 

-.047 

.072 

.850 

7 

.750 

.893 

.548* 

.321 

.393 

.510 

-.143 

.072 

.641 

8 

1.000 

1.000 

.000 

.286 

.607 

.535 

.000 

.321 

.741 

9 

.500 

.857 

-.032 

.071 

.179 

.309 

-.357 

.108 

.917 

10 

.500 

.500 

.000 

.000 

.214 

.309 

.000 

.214 

1.000 

11 

1.000 

.964 

.304 

.321 

.143 

.015 

.036* 

-.178 

.667 

12 

1.000 

.929 

.132 

.286 

.429 

.459 

.071* 

.143 

.692 

13 

.500 

.821 

. 737** 

.214 

.214 

.357 

-.321 

.000 

.739 

14 

1.000 

.857 

.420 

.607 

.464 

.630* 

.143* 

-.143 

.292 

15 

.500 

.679 

.580** 

.143 

.214 

.357 

-.179 

.071 

.789 

16 

1.000 

1.000 

.000 

.143 

.500 

.535 

.000 

.357 

.857 

17 

1.000 

1.000 

.000 

.393 

.964 

.394 

.000 

.571 

.607 

13 

.875 

.964 

.304 

.679 

.786 

.397 

-.089 

.107 

.296 

1 


♦ p < .05 


**  p<  .01 
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b.  The  discrimination  index  used  for  both  BDI  and  PDI  is  the  phi -coefficient. 

In  the  case  of  BDI,  all  students  with  scores  of  four  or  more  Items  correct  on  the 
pretest  were  classified  into  the  upper  criterion  group,  and  all  other  students  were 
classified  into  the  lower  criterion  group.  In  the  case  of  PDI,  all  students  with  scores 
of  15  or  more  items  correct  on  the  posttest  were  classified  into  the  upper  group,  and 
all  students  with  scores  of  12  or  fewer  items  correct  were  classified  into  the  lower 
group.  Both  BDI  and  PDI  were  tested  using  a correction  for  discontinuity  (See 
Edwards,  1967,  p.  333. ) and  two-tailed  probability  levels. 

c.  The  categorical  error  rates  and  discrimination  indices  given  in  Table  3 are 
based  upon  the  cut-off  values  indicated  in  the  footnotes  to  that  table.  The  cut-offs 
used  were  selected  primarily  for  illustrative  purposes,  and  are  not  necessarily  in- 
tended to  be  optimal  cut-offs  from  a theoretical  standpoint.  Note  that,  while  not 
necessary,  the  cut-offs  used  are  the  same  for  all  items.  (The  reader  may  find  it 
instructive  to  reanalyze  the  data  with  different  cut-offs. ) 
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TABLE  3.  Categorical  Error  Rates  and  Discrimination  Indices  for  a CAI  Program  in  Microeconom'cs 


Raw  Error  Rates  and  Discrimination  Indicesa 

Derived 
Error  Rates 

Objective 

Number 

TER 

BER 

BDI 

IER 

PER 

PDI 

DERb 

RER° 

PMPGd 

1 

H 

H 

0 

L 

L 

0 

♦ 

- 

- 

2 

H 

H 

0 

L 

H 

+ 

- 

GT 

- 

3 

H 

H 

0 

L 

L 

0 

- 

- 

- 

4 

H 

H 

♦ 

L 

L 

0 

- 

- 

jr 

5 

H 

H 

+ 

H 

L 

0 

- 

LT 

LT 

6 

H 

H 

' + 

L 

L 

0 

- 

- 

I 

7 

H 

H 

+ 

H 

H 

0 

- 

- 

- 

8 

H 

H 

0 

L 

H 

0 

- 

GT 

- 

9 

H 

H 

0 

L 

L 

0 

- 

- 

- 

10 

H 

H 

0 

L 

L 

0 

- 

GT 

' 

11 

H 

H 

0 

H 

L 

0 

* 

- 

1 

12 

H 

H 

0 

L 

H 

0 

♦ 

- 

13 

H 

H 

+ 

L 

L 

0 

- 

- 

- 

14 

H 

H 

0 

H 

H 

+ 

♦ 

- 

LT 

15 

H 

H 

+ 

L 

L 

0 

- 

- 

- 

16 

H 

H 

0 

L 

H 

0 

- 

GT 

17 

H 

H 

0 

H 

H 

0 

- 

GT 

- 

18 

H 

H 

0 

H 

H 

0 

- 

- 

LT 
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NR(6)  NR(6)  NR(6)  NR(12)  NR(6)  NR(6) 


d.  Table  4 lists  the  decisions  that  result  from  applying  the  various  decision  rules 
to  three  different  subsets  of  the  data  reported  in  Table  3.  When  two  rules  indicate  a 
need  for  revision,  both  are  given;  in  most  other  cases,  only  one  rule  is  applicable. 
Occasions  do  arise,  however,  when  two  or  more  different  decisions  are  applicable  to 
the  same  item  or  segment  of  instruction.  For  example,  objective  number  five  has 
IER  = H,  PER  = L,  and  PDI  = 0.  According  to  Rule  6 the  instruction  does  not  need 
revisions,  but  Rule  14  indicates  that  the  instruction  is  questionable.  We  have  chosen 
to  resolve  such  conflicts  by  selecting  the  decision  that  has  the  most  serlcas  implica- 
tions for  revision;  i.e.,  "questionable"  (?)has  more  serious  implications  for  re- 
vision than  "do  not  revise"  (NR),  and  "revise"  (R)  has  more  serious  implications  for 
revision  than  either  "questionable"  (?)  or  "do  not  revise"  (NR).  Thus,  for  objective 
number  five  we  have  labelled  the  instruction  "questionable"  in  the  second  set  of 
decisions. 

e.  In  Table  4 the  first  set  of  decisions  uses  more  data  than  the  second  which,  in 
turn,  uses  more  data  than  the  third.  One  possible  effect  of  decreasing  the  amount  of 
data  used  is  illustrated  by  the  decisions  with  regard  to  instruction  for  objective  num- 
ber five.  Using  all  of  the  data  for  objective  five  in  Table  4,  Rule  19  indicates  that 
the  instruction  should  be  revised.  When,  however,  derived  erro  - rates  are  elimin- 
ated, Rule  19  becomes  inapplicable,  and  Rule  14  indicates  that  the  instruction  should 
be  examined,  but  not  necessarily  revised.  Finally,  when  both  derived  error  rates 
and  IER  are  eliminated,  both  Rules  19  and  14  become  inapplicable,  and  Rule  6 in- 
dicates that  no  revision  is  required.  This  situation  is  an  empirical  demonstration  of 
the  desirability  of  obtaining  as  much  data  as  possible  in  order  to  strengthen  decisions 
about  the  adequacy  of  instruction. 

f.  This  statement  does  not,  however,  imply  that  an  increase  in  the  amount  of 
available  data  will  necessarily  increase  the  number  of  decisions  involving  the  re- 
vision (R)  of  items  or  instruction.  Consider,  for  example,  the  decisions  involving 
instruction,  given  in  Table  4,  for  objective  number  11.  Using  only  pretest  and  post- 
test data,  no  revision  is  required  according  to  Rule  6 . When  IER  is  included  as  data 
for  decision  making,  the  second  set  of  decisions  indicate  that  the  instruction  is  ques- 
tionable according  to  Rule  14.  When,  however,  all  available  data  are  used  (i.e., 
pretest  and  posttest  data,  IER,  and  the  derived  emirates),  we  again  arrive  at  the 
decision  "no  revision"  according  to  Rules  6 and  18.  Clearly,  in  the  case  of  ob- 
jective 11,  an  increase  in  the  amount  of  available  data  ultimately  confirms  our  initial 
judgment  that  no  revision  of  instruction  is  required. 

g.  For  this  particular  instructional  system.  Table  4 indicates  that  the  avail- 
ability of  derived  error  rates  increases  the  number  of  decisions  that  involve  revision 
of  items  and  instruction.  Furthermore,  in  general,  revision  is  most  often  neces- 
sitated by  relatively  poor  performance  on  the  posttest  (note  the  many  times  Rules  8 
and  9 are  employed)  and  relatively  poor  retention  (note  the  many  times  Rules  13  and 
17  are  employed).  Also,  the  instruction  seems  to  be  in  more  need  of  revision  than 
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the  test  items.  These  general  observations  do,  in  fact,  coincide  with  the  predictions 
of  the  person  responsible  for  developing  this  particular  instructional  program. 11 

h.  The  following  example  empirically  illustrates  the  contribution  of  various  types 
of  data  to  the  decision  process;  however,  there  are  clearly  more  efficient  ways  to 
evaluate  instruction,  in  the  formative  sense  of  the  term,  using  the  rationale,  types 

of  data,  and  decision  rules  previously  presented.  Consider,  for  example,  the  data 
presented  in  Table  5. 

i.  These  data  reflect  the  performance  of  39  students  on  test  questions  related  to 

12  terminal  objectives  selected  from  a multimedia  economics  course  during  its  field- 
testing  phase. i.  2 The  first  11  objectives  were  tested  by  a pretest,  terminal,  and 
posttest  item;  the  last  objective  was  tested  by  a pretest  and  posttest  item,  but  not  a 
terminal  item. 


TABLE  5.  Data  and  Decisions  for  Selected  Objectives  from  a Multimedia  Economics  Course 


i 

j.  The  reader  can  easily  verify  that  the  six  types  of  data  reported  in  Table  5 are 
a sufficient  set  in  terms  of  the  Rules  listed  in  Table  1 (recall  that  Rules  16-18  can  be 
used  to  replace  Rules  1-3  and  12-15).  For  the  decisions  reported  in  Table  5,  the 
high/low  cut-off  value  for  PER  is  0.30,  Cj  is  0.20,  c2  is  0.25,  and  c3  is  0.70.  The 
B index  (See  Brenuan,  in  press.)  was  used  for  both  BDI  and  PDI. 

k.  Using  the  decisions  indicated  in  Table  5 we  referred  back  to  the  instructional 
material  in  search  of  explanations  for  the  indicated  "problems".  For  objectives  1, 

3,  5,  and  10  we  could  eliminate  at  least  two  distractors  in  the  associated  pretest  items 
without  reading  the  subsequent  instructional  material.  The  pretest  items  for  objectives 
6 and  7 involved  the  interpretation  of  very  complex  graphs;  however,  this  activity 
seems  to  be  taught  within  the  course,  as  evidenced  by  posttest  performance  for  these 
objectives.  For  objectives  2 and  9,  which  involve  mathematical  or  graphical  skills, 
the  retention  problem  appears  to  be  the  primary  difficulty.  The  terminal  and  posttest 
items  for  objective  4 seem  to  be  different  enough  to  warrant  revision  of  at  least  one  of 
them.  Objective  8 seems  to  be  achieved  after  the  terminal  item  for  this  objective  is 
presented.  Objectives  11  and  12  seem  to  require  further  instruction. 

l.  The  objectives  discussed  above  are  only  a very  small  fraction  of  the  total  num- 
ber of  objectives  covered  in  this  course;  in  the  total  course  97  percent  of  the  students 
achieved  80  or  more  percent  of  the  objectives.  Nevertheless,  it  appears  that  the  pro- 
cedure we  have  outlined  can  serve  a useful  function  for  formative  evaluation. 

Discussion 

a.  It  is  certainly  reasonable  to  expect  that  some  readers  may  feel  that  certain 
decisions  we  have  proposed  are  not  appropriate  for  their  particular  programs,  or 
that  other  decision  rules  should  be  added.  We  have  tried  to  specify  those  decisions 
that  we  feel  are  the  most  universally  applicable;  however,  even  more  important  than 
the  actual  decision  rules  presented  is  the  method  used  to  arrive  at  decisions  about 
test  items  and  instruction.  This  method  appears  to  be  generalizable  to  a wide  variety 
of  instructional  situations,  and  it  seems  to  maximize  the  use  made  of  instructlonally 
relevant  data  heretofore  used  separately  rather  than  as  elements  of  a coherent  system 
of  analysis  for  decision. 

b.  In  this  section  we  will  discuss  various  factors  that  have  applicability  to  the 
rules  we  have  presented  and  the  decision  process  we  have  proposed. 

Instructional  Systems  and  Criterion-Referenced  Testing 

a.  One  might  define  an  Instructional  system  in  general  as  a replicable  method  of 
instruction  providing  feedback  that  can  be  used  for  revision  purposes.  Replicability 
is  always  a matter  of  degree,  but  particular  approaches  appear  to  differ  on  the  aver- 
age in  the  respect;  e.g. , instruction  provided  by  teachers  is  less  replicable  than  that 
provided  by  a computer.  Also,  in  addition  to  replicability,  instructional  systems  are 
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usually  characterized  by  a close  correspondence  between  test  items  and  behavioral 
objectives,  i.e.,  test  items  are  criterion -referenced.  In  addition,  it  is  usually  ex- 
pected that  "most"  of  the  students  will  get  "most"  of  the  terminal  and  posttest  items 
correct . 

b.  Brennan  (1970)  and  Popham  & Husek  (1969)  have  examined  some  aspects  of 
the  applicability  of  classical  test  theory  to  the  analysis  of  criterion -referenced  tests. 
Perhaps  the  most  important  implication  of  these  analyses  is  that  the  classical  normal- 
ity assumptions  concerning  errors  of  measurement  do  not  seem  to  be  appropriate  in 
many  criterion -referenced  testing  situations;  the  errors  of  measurement  seem  to  be 
better  characterized  by  binomial  error  models.  (See  Lord  & Novick,  1968,  Chapter 
23.)  This  means  that  many  of  the  statistics  used  in  classical  test  theory  are  not  ap- 
plicable in  the  criterion -referenced  testing  situation.  For  example,  the  biserial 
discrimination  index  is  not  appropriate  for  criterion-referenced  test  data,  since  total 
scores  on  the  test  are  not  necessarily  normally  distributed;  a similar  comment  can  be 
made  about  the  tetrachoric  discrimination  index. 

c.  Another  characteristic  of  a good  instructional  system  is  that  all  students  who 
possess  the  necessary  prerequisites  and  who  receive  instruction  should  achieve 
criterion  performance  on  the  posttest.  (See  Stolurow  & Davis,  1965.)  Ideally,  in 
fact,  we  may  want  all  students  to  achieve  all  objectives.  In  such  a situation  all  items 
would  be  nondiscriminating  (assuming,  of  course,  that  total  test  score  is  the  criterion 
used  for  judging  discriminability).  This  line  of  reasoning  indicates  why  we  have  spe- 
cified that  nondiscriminating  items  do  not  indicate  a need  for  revision.  Conversely, 
items  that  are  significantly  discriminating  (especially  negatively  discriminating  items) 
indicate  a possible  need  for  revision  since  the  instructional  system  is  performing 
worse  for  one  group  of  students  than  for  another  group. 

Corresponding  Items 

a.  When  discussing  the  context  of  the  rationale  that  has  been  presented,  we 
assumed  that  for  each  objective  there  exists  a pretest,  posttest,  and  terminal  test 
item;  furthermore,  we  assumed  that  the  items  testing  a given  objective  are,  in  some 
sense,  "corresponding",  "equivalent",  or  "parallel". 

b.  The  terms  "equivalent"  and  "parallel"  are,  in  the  classical  sense,  usually 
applied  to  tests.  A set  of  k tests  are  said  to  be  "parallel"  or  "equivalent"  if  they 
have  equal  means,  equal  variances,  and  equal  intercorrelations.  (See  Gulliksen, 

1950,  p.  173.)  This  does  not  mean,  however,  that  there  is  necessarily  any  strict 
correspondence  among  items  in  the  k tests.  Thus,  in  the  rationale  that  we  have  pro- 
posed, and  in  criterion-referenced  testing  in  general,  the  classical  concept  of  parallel 
tests  is  clearly  not  sufficient,  since  we  are  very  concerned  about  the  performance  of 
students  on  individual  items,  not  just  entire  tests.  Let  us,  therefore,  reserve  the 
terms  "parallel"  and  "equivalent"  for  entire  tests,  and  examine  the  analogous  issue 

of  ’ corresponding"  items. 
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c.  We  can  define  "corresponding"  items,  in  general,  as  items  that  measure  the 


same  thing.  Clearly,  then,  one  requirement  of  corresponding  items  is  that,  in  the 
judgment  of  specialists  the  items  measure  the  same  behavioral  objective.  Further- 
more, just  as  we  have  a statistical  criterion  for  parallel  tests,  it  seems  reasonable 
to  propose  a similar  statistical  criterion  for  corresponding  items.  Thus,  we  might 
propose  that  corresponding  items  have  equal  means,  equal  variances,  and  equal  inter- 
correlations. Since  we  are  assuming  that  items  are  scored  dichotomously,  the  mean 
of  item  i is  simply  the  proportion  of  correct  responses  (p^)  and  the  correlation  be- 
tween any  two  corresponding  items  is  the  phi  correlation  (r^). 

d.  Now,  suppose  we  give  a set  of  k tests  to  N students  in  order  to  determine 
whether  or  not  the  tests  are  parallel;  i.e. , whether  or  not  the  set  of  k means,  k vari- 
ances, and  k(k  - l)/2  intercorrelations  are  equal  except  for  sampling  differences. 
Wilks  (1946)  provides  a statistical  test  to  answer  this  question. 

e.  Unfortunately,  however,  Wilks'  test  is  not  applicable  for  judging  the  equality 
of  a set  of  means,  variances,  and  intercorrelations  for  k dichotomously  scored 
criterion -referenced  items.  Wilks'  test  assumes  a normal  multivariate  population 
distribution,  and,  as  we  have  stated  previously,  the  assumption  of  normality  is  prob- 
ably inappropriate  in  most  criterion -referenced  testing  situations. 

f.  As  far  as  we  know,  there  is  no  currently  available  method  for  simultaneously 
testing  the  equality  of  means,  variances,  and  correlations  among  dichotomously 
scored  items  that  are  not  necessarily  normally  distributed.  We  can,  however, 
approach  a solution  to  the  problem  by  applying  what  is  usually  called  Cochran's  Q 
Test  (See  Siegel,  1956,  pp.  161-166.),  which  is  a test  for  the  equality  of  means,  or 
proportions  (pj),  among  dichotomized  variable  (in  this  case,  test  items). 

g.  Since  the  variance  of  a dichotomous  variable  scored  zero  or  one  is  completely 
determined  by  the  mean  (or  proportion  of  successes),  it  is  clear  that  if  the  means  of 
k items  are  equal,  then  the  variances  will  also  be  equal.  However,  even  if  the  means 
and  variances  of  k items  are  equal  (except  for  sampling  differences),  this  does  not 
necessarily  mean  that  the  intercorrelations  are  equal.  The  authors  have  no  knowledge 
of  any  currently  available  method  to  test  the  equivalence  of  intercorrelations  (phi- 
coefficients)  among  dichotomously  scored  items  which  may  not  be  distributed  normally 
in  the  population. 

h.  Besides  the  problem  of  nonnormally  distributed  variables  there  is  another 
problem  in  testing  the  equivalency  of  intercorrelations  (phi-coefficients)  that  may  not 
be  immediately  evident.  Suppose  we  have  three  items  (i.e. , k = 3).  In  order  to  test 
whether  or  not  the  intercorrelations  among  the  items  are  the  same,  we  must  take  into 
account  three  different  phi -coefficients:  (a)  r^  between  item  one  and  item  two,  (b) 

r^  between  item  one  and  item  three,  and  (c)  r^  between  item  two  and  item  three. 

Now,  it  is  clear  that  (a)  and  (b)  are  correlated  because  both  phi -coefficients  are  based 
on  the  same  data  for  item  one;  (a)  and  (c)  are  also  correlated  since  they  are  based  on 
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the  data  for  item  two;  an  J finally,  (b)  and  (c)  are  correlated  becauee  they  are  baaed 
on  the  same  data  for  item  three.  Sb.ce  the  three  r^'s  are  clearly  correlated,  we  can- 
not apply  any  of  the  wel.  -known  chi-square  testa  that  are  currently  available  for  use 
with  contingency  tables.  In  the  absence  of  a test  of  significance  for  examining  the 
equivalence  of  intercor-  elations  (phi -coefficients)  among  k items,  the  evaluator  will 
probably  have  to  use  his  best  judgment  about  whether  or  not  the  phi -coefficients  are 
"approximately"  equal. 

i.  In  summary,  we  nave  defined  corresponding  items  as  items  that  (a)  measure 
the  same  behavioral  objective,  (b)  have  the  same  means,  (c)  have  the  same  variances, 
and  (d)  have  the  same  intercor relations.  We  have  recommended  Cochran’s  Q Test  as 
a method  for  testing  (b)  and  (c).  The  lack  of  an  available  statistical  test  for  (d)  may 
not  be  too  serious  a limitation;  certainly,  if  conditions  (a),  (b),  and  (c)  are  fulfilled 
and  the  intercorrelations  among  the  items  arc  approximately  equal,  it  is  reasonable 
to  assume  that  the  item0  are  "corresponding". 

Comments  on  Data  for  decision  Making 

a.  For  purposes  of  simplicity,  the  decision  rules  we  have  specified  are  based 
upon  data  from  one  pretest  item,  one  terminal  item,  and  one  posttest  item  for  each 
objective.  There  may,  of  course,  be  more  than  one  pretest,  terminal,  and^>r  post- 
test item  for  any  given  objective.  Such  additional  data  can  be  taken  into  account  in 
various  ways.  For  example,  one  might  merely  combine  the  data  from  all  the  pretest 
(posttest  or  terminal)  items  relevant  to  a given  objective  in  order  to  calculate  the 
appropriate  error  rate.  Alternatively,  assuming,  for  example,  that  three  posttest 
items  test  the  same  terminal  objective,  one  might  specify  that  if  a student  answers 
two  of  the  three  items  correctly,  then  he  has  achieved  the  objective.  Other  alterna- 
tives are  also  possible;  however,  a multiplicity  of  pretest,  posttest,  or  terminal 
items  relevant  to  a given  objective  can  complicate  the  interpretation  of  which  item, 

if  any,  requires  revision. 

b.  We  have  also  assumed  that  every  student  answers  every  item.  There  are 
several  formulas  available  (See  Guilford,  1954,  pp.  418-424. ) that  can  be  used  to 
calculate  error  rates  vith  missing  data.  Such  formulas  can  be  used  instead  of 
Equation  2.  A large  amount  of  missing  data  can,  however,  present  serious  prob- 
lems, especially  if  the  sample  size  is  small. 

c . There  are  many  discrimination  indices  available  in  the  literature  (See 
Guilford,  1954,  pp.  424-440. ) than  could  be  used  to  calculate  BDI  and  PDI.  In  our 
opinion,  however,  the  phi -coefficient  and  the  B index  (See  Brennan  1970,  1972.)  are 
the  best  indices  to  use  with  criterion -referenced  tests,  since  they  make  only  weak 
distributional  assumptions,  and  they  allow  the  evaluator  to  specify  virtually  any  cut- 
off between  upper  and  lower  groups.  In  addition,  the  index  B has  a very  useful 
interpretation  in  terms  of  the  number  of  discriminations  made  by  an  item. 
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d.  One  further  comment  seems  appropriate.  Stolurow  and  Frincke  (1966)  have 
noted  that  there  is  a danger  of  rejecting  good  items  (or  good  instruction)  when  the 
sample  size  is  relatively  small,  say  N = 15  or  20.  In  their  study,  Stolurow  and 
Frincke  were  concerned  about  error  rates  only.  Since,  in  this  paper  we  examine 
both  error  rates  and  discrimination  indices,  it  is  certainly  desirable  that  the  sample 
size  be  adequate  for  most  purposes . The  technique  we  have  proposed  can  be  used 
with  smaller  sample  sizes;  however,  the  certainty  with  which  decisions  can  be  made 
is  thereby  reduced. 
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FOOTNOTES 


Much  of  the  research  reported  herein  was  performed  pursuant  to  con- 
tracts with  the  Unived  States  Naval  Academy,  Contract  No.  N00161-70-C-0119,  and 
the  Office  of  Naval  Research,  Contract  No.  N00014-67-A-0298-0032. 

2 Items  that  have  a virtual  infinitude  of  possible  answers  have  TER  = 1.00; 
however,  the  evaluator  should  be  careful  not  to  assume  that  every  free-response  or 
open  ended  test  item  has  TER  = 1.00.  Very  often  such  items  are  so  worded  that  only 
two  or  three  answers  are  really  possible,  in  which  case  TER  = 0.50  or  TER  * 0.67. 

3 

Terminal  Error  Rate  would  be  a more  descriptive  phrase  than  Instruc- 
tional Error  Rate;  however,  we  have  chosen  the  latter  to  avoid  the  ambiguity  involved 
in  having  TER  stand  for  both  Terminal  Error  Rate  and  Theoretical  Error  Rate. 

^ R is  also  possible  that  the  item  has  neither  of  these  faults  and  the  ob- 
jective, while  being  easy  for  most  of  the  students,  is  considered  to  be  an  integral 
part  of  the  total  set  of  objectives.  In  this  case,  the  item  would  not  be  revised.  A 
similar  statement  can  be  made  for  Rule  16  which  will  be  discussed  later. 

^ It  is  also  possible  that  the  terminal  item  (frame)  and  the  posttest  item 
are  not  measuring  the  same  content  at  the  same  level  of  difficulty,  even  though  this 
is  an  assumption  underlying  all  the  decision  rules  presented  here.  Also,  if  students 
receive  confirmation  on  terminal  items  within  the  instructional  system,  then  one 
could  argue  that  the  occurrence  of  IER  = H and  PER  = L does  not  indicate  any  need 
for  revision. 

6 The  term  - 1/(2N)  in  Equation  7 is  a correction  for  discontinuity  and, 
as  such,  can  be  dropped  if  the  sample  size  is  large.  Note  that  when  TER  = 1.00  Z 
is  undefined;  in  this  case  any  value  of  DER  > 0 can  be  considered  significant. 

7 

Ideally,  in  order  to  choose  cut-offs  for  RER  parametric  data  are  needed 
from  a variety  of  situations  to  determine  the  mean  ,md  standard  deviation  of  retention 
scores  in  relation  to  the  investment  in  learning.  One  interim  possibility  is  to  use 
the  principle  of  homogeneity  in  relation  to  comparable  data  within  a program.  The 
grossly  deviant  retention  scores  would  reveal  the  items  in  greatest  need  of  revision. 
For  example,  the  cut-off  could  be  the  n most  extreme  items  where  n is  specified  by 
using  a percentage  (e.g. , 5 or  10  percent)  of  the  total  number  of  items. 

® One  could  make  a case  for  using  error  rate  on  the  posttest  item  (PER) 
rather  than  error  rate  on  the  terminal  item;  then,  however,  PER  - BER  would 
involve  a confounding  of  gain  with  retention,  as  we  are  using  the  terms  in  this  paper. 
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We  are  grateful  to  Mr.  Eugene  Millsteln  for  developing  the  instructional 
program  and  collecting  the  data  pursuant  to  a contract  with  the  Office  of  Naval  Re- 
search, ONR  Contract  No.  N00014-67-A-0298-0003  conducted  at  the  Harvard 
Computer-Aided  Instruction  Laboratory.  Our  analysis  of  the  data  should  not,  however, 
be  interpreted  as  an  evaluation  of  the  program.  (See  Millstein,  1971.) 

10  Recall  that  when  derived  error  rates  are  available,  Rules  16-19  replace 
Rules  1-3  and  12-15,  since  the  former  rules  are  more  exact  statements  oi  the  latter 
rules.  More  specifically.  Rule  18,  in  effect,  replaces  Rule  14.  For  objective  number 
11,  application  of  Rule  18  indicates  no  need  for  revision,  which  overrides  the  decision 
made  on  the  basis  of  Rule  14. 

The  reader  will  note  that,  in  Table  4,  if  two  or  more  rules  indicate 
"ho  revision  (NR)",  we  have  identified  only  that  rule  which  we  believe  is  most  impor- 
tant. There  seems  to  be  no  particular  advantage  in  identifying  all  the  possible  reasons 
for  doing  nothing! 

11  This  program  is  being  used  primarily  as  a vehicle  for  testing  a psycho- 
logical theory  of  sequencing  instruction.  As  such,  the  program  has  been  purposely 
written  to  discriminate  among  students  who  have  experienced  different  instructional 
sequences;  the  program  is  not  meant  to  teach  microeconomics  to  all  students  in  the 
most  effective  manner. 

12  We  are  grateful  to  Sterling  Institute  of  Washington,  D.  C. , the  developer 
of  this  course,  for  making  these  data  available  to  us.  However,  we  wish  to  emphasize 
that  the  data  reported  and  all  explanations  of  such  data  are  incomplete  and  are  used 
here  for  illustrative  purposes  only;  in  no  way  should  our  analysis  of  these  data  be 
construed  as  an  evaluation  of  their  course. 

a The  cut-off  value  for  TER,  EER,  IER  and  PER  is  0.30. 

b indicates  that  DER  is  not  significantly  greater  than  zero  at  the  . 05 
level  for  a one-tailed  test  of  significance. 

c iiQ-pn  indicates  that  RER  is  "greater  than"  0.20. 

"LT"  indicates  that  RER  is  'less  than"  -0.30. 
indicates  that  -0.30  & RER  ± 0.20. 

d "lt"  indicates  that  PMPG  is  'less  than"  0.60. 

"-"  indicates  that  PMPG  is  greater  than  or  equal  to  0.60. 

* p <<  . 05 
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Appendix 


CIVILIAN  EDUCATIONAL  CONSULTANT  ROSTER 
Dr.  O.  Dennis  Barnes 

Chairman,  Department  of  Instructional  Technology 
Rochester  Institute  of  Technology 
One  Lomb  Memorial  Drive 

Rochester,  New  York  14623  (716)  464-2892 

Ms . Jan  Danford 
Director  of  Curriculum 
Capital  Area  Career  Center 

Mason,  Michigan  (517)  676-1051 

*Dr.  Alexander  Schure 
Director,  External  Degree  Programs 
Wheatly  Road 

Old  Westbury,  New  York  11568  (515)  626-3910 

Dr.  William  F.  Fitzgerald 

Director,  Div.  of  Educational  Resources  and  Studies 
American  Association  of  Dental  Schools 
1625  Massachusetts  Avenue  N.  W. 

Washington,  D.  C.  20036  (313)  763-1031 

Brigadier  General  Richard  C.  Horne,  m (Ret) 

100  Country  Club  Court 

Williamsburg,  Virginia  23185  (804)  229-9608  (home) 

Dr.  Harold  E.  Mitzel 
Associate  Dean  for  Research 
The  Pennsylvania  State  University 
277  Chambers  Building 

University  Park,  Pennsylvania  16802  (814)  865-2525 

Mrs.  Catherine  E.  Morgan 
Director,  CAI  Program 
Einstein  High  School 
11135  Newport  Mill  Road 

Kensington,  Maryland  20795  (301)  949-8700 

Mr.  Tom  O'Sullivan 

Training  Requirements  Analysis  Co.  (TRAC) 

1220  Sunset  Plaza  Drive 

Los/ngeles,  California  90069  (213)  652-2344 
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GOVERNMENT  CONSULTANT  ROSTER 


COL  William  Luebbert 
Associate  Professor  and  Director 
Instruction  Support  Division 
US  Military  Academy 

West  Point,  New  York  10996  AUTOVON  688-4744 

LTC  Allan  B.  Salisbury 
Systems  Engineering  Directorate 
OPM  ARTADS 

Fort  Monmouth,  New  Jersey  07703  AUTOVON  995-2761 

Dr.  Marty  Rockway 
Technical  Director 

Air  Force  Human  Resources  Laboratory 
Lowry  Air  Force  Base 

Denver,  Colorado  80232  AUTOVON  926-4385 

Dr.  John  D.  Ford,  Jr. 

Deputy  Director 

CAI  Research  Department 

Naval  Personnel  Research  Activity 

San  Diego,  California  92152  AUTOVON  933-7121 

Dr.  Harold  O'Neill,  Jr. 

Advanced  Research  Projects  Agency 
Human  Resources  Research  Office 
1400  Wilson  Boulevard 

Arlington,  Virginia  22209  AUTOVON  224-1304 

LTC  Robert  W.  Otto 
HQDA  (DACS-DIP) 

Washington,  DC  20310  AUTOVON  225-0041 

Mr.  William  Lundy 

US  Army  Missile  and  Munitions  Center  and  School 
ATTN:  ATSK-TEA-C 

Redstone  Arsenal,  Alabama  35809  AUTOVON  746-2779 

Mr.  Frank  Dare 

US  Army  Ordnance  Center  and  School 
ATTN:  ATSOR-1 

Aberdeen  Proving  Grounds,  Maryland  21005  AUTOVON  870-5327 


Dr.  Robert  J.  Seidel 
Program  Director 
Instructional  Technology  Group 
Human  Resources  Research  Organization 
300  North  Washington  Street 

Alexandria,  Virginia  22314  (703  ) 549-3611 

Dr.  Lawrence  M.  Stolurow 

Ex.  Director,  Institute  for  Research  in  Learning  and  Instruction 
State  University  of  New  York  at  Stony  Brook 

Stony  Brook,  New  York  11794  (516)  246-7061/6730 

Ms.  Ruann  Pengov 

Ohio  State  University,  College  of  Medicine 

Div.  of  Computer  Sciences  for  Medical  Education  and  Research 

Health  Sciences  Laboratory 

376  West  10th  Street 

Columbus,  Ohio  43210  (614)  422-6192 


*Dr.  Ward  Deutschman  attended  on-site  meeting  in  place  of  Dr.  Alexander 
Schure . 
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